=== RUN TestStartStop/group/old-k8s-version/serial/SecondStart
start_stop_delete_test.go:256: (dbg) Run: out/minikube-linux-arm64 start -p old-k8s-version-929157 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --container-runtime=containerd --kubernetes-version=v1.20.0
E0305 04:34:41.209156 2407150 cert_rotation.go:168] key failed with : open /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/addons-015258/client.crt: no such file or directory
start_stop_delete_test.go:256: (dbg) Non-zero exit: out/minikube-linux-arm64 start -p old-k8s-version-929157 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --container-runtime=containerd --kubernetes-version=v1.20.0: exit status 102 (6m9.354602294s)
-- stdout --
* [old-k8s-version-929157] minikube v1.32.0 on Ubuntu 20.04 (arm64)
- MINIKUBE_LOCATION=17909
- MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
- KUBECONFIG=/home/jenkins/minikube-integration/17909-2401808/kubeconfig
- MINIKUBE_HOME=/home/jenkins/minikube-integration/17909-2401808/.minikube
- MINIKUBE_BIN=out/minikube-linux-arm64
- MINIKUBE_FORCE_SYSTEMD=
* Kubernetes 1.28.4 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.28.4
* Using the docker driver based on existing profile
* Starting "old-k8s-version-929157" primary control-plane node in "old-k8s-version-929157" cluster
* Pulling base image v0.0.42-1708944392-18244 ...
* Restarting existing docker container for "old-k8s-version-929157" ...
* Preparing Kubernetes v1.20.0 on containerd 1.6.28 ...
* Verifying Kubernetes components...
- Using image fake.domain/registry.k8s.io/echoserver:1.4
- Using image docker.io/kubernetesui/dashboard:v2.7.0
- Using image registry.k8s.io/echoserver:1.4
- Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Some dashboard features require the metrics-server addon. To enable all features please run:
minikube -p old-k8s-version-929157 addons enable metrics-server
* Enabled addons: storage-provisioner, dashboard, metrics-server, default-storageclass
-- /stdout --
** stderr **
I0305 04:34:20.583845 2604473 out.go:291] Setting OutFile to fd 1 ...
I0305 04:34:20.584086 2604473 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0305 04:34:20.584114 2604473 out.go:304] Setting ErrFile to fd 2...
I0305 04:34:20.584133 2604473 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0305 04:34:20.584421 2604473 root.go:338] Updating PATH: /home/jenkins/minikube-integration/17909-2401808/.minikube/bin
I0305 04:34:20.584829 2604473 out.go:298] Setting JSON to false
I0305 04:34:20.585944 2604473 start.go:129] hostinfo: {"hostname":"ip-172-31-30-239","uptime":37006,"bootTime":1709576255,"procs":249,"os":"linux","platform":"ubuntu","platformFamily":"debian","platformVersion":"20.04","kernelVersion":"5.15.0-1055-aws","kernelArch":"aarch64","virtualizationSystem":"","virtualizationRole":"","hostId":"92f46a7d-c249-4c12-924a-77f64874c910"}
I0305 04:34:20.586042 2604473 start.go:139] virtualization:
I0305 04:34:20.593393 2604473 out.go:177] * [old-k8s-version-929157] minikube v1.32.0 on Ubuntu 20.04 (arm64)
I0305 04:34:20.595562 2604473 out.go:177] - MINIKUBE_LOCATION=17909
I0305 04:34:20.595616 2604473 notify.go:220] Checking for updates...
I0305 04:34:20.600360 2604473 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0305 04:34:20.602695 2604473 out.go:177] - KUBECONFIG=/home/jenkins/minikube-integration/17909-2401808/kubeconfig
I0305 04:34:20.604366 2604473 out.go:177] - MINIKUBE_HOME=/home/jenkins/minikube-integration/17909-2401808/.minikube
I0305 04:34:20.606408 2604473 out.go:177] - MINIKUBE_BIN=out/minikube-linux-arm64
I0305 04:34:20.608208 2604473 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0305 04:34:20.610934 2604473 config.go:182] Loaded profile config "old-k8s-version-929157": Driver=docker, ContainerRuntime=containerd, KubernetesVersion=v1.20.0
I0305 04:34:20.613780 2604473 out.go:177] * Kubernetes 1.28.4 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.28.4
I0305 04:34:20.616620 2604473 driver.go:392] Setting default libvirt URI to qemu:///system
I0305 04:34:20.661217 2604473 docker.go:122] docker version: linux-25.0.3:Docker Engine - Community
I0305 04:34:20.661326 2604473 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0305 04:34:20.788064 2604473 info.go:266] docker info: {ID:6ZPO:QZND:VNGE:LUKL:4Y3K:XELL:AAX4:2GTK:E6LM:MPRN:3ZXR:TTMR Containers:2 ContainersRunning:1 ContainersPaused:0 ContainersStopped:1 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:53 OomKillDisable:true NGoroutines:71 SystemTime:2024-03-05 04:34:20.773178045 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1055-aws OperatingSystem:Ubuntu 20.04.6 LTS OSType:linux Architecture:aar
ch64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:2 MemTotal:8215040000 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ip-172-31-30-239 Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=builtin] ProductLicense: Warnings:<nil> ServerErro
rs:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.6]] Warnings:<nil>}}
I0305 04:34:20.788178 2604473 docker.go:295] overlay module found
I0305 04:34:20.790449 2604473 out.go:177] * Using the docker driver based on existing profile
I0305 04:34:20.792280 2604473 start.go:297] selected driver: docker
I0305 04:34:20.792301 2604473 start.go:901] validating driver "docker" against &{Name:old-k8s-version-929157 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.0 ClusterName:old-k8s-version-929157 Namespace:default APIServerHAVIP: AP
IServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.20.0 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[dashboard:true default-storageclass:true metrics-server:true storage-provisioner:true] CustomAddonImages:map[MetricsScraper:registry.k8s.io/echoserver:1.4 MetricsServer:registry.k8s.io/echoserver:1.4] CustomAddonRegistries:map[MetricsServer:fake.domain] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:fa
lse MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0305 04:34:20.792420 2604473 start.go:912] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0305 04:34:20.793003 2604473 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0305 04:34:20.912696 2604473 info.go:266] docker info: {ID:6ZPO:QZND:VNGE:LUKL:4Y3K:XELL:AAX4:2GTK:E6LM:MPRN:3ZXR:TTMR Containers:2 ContainersRunning:1 ContainersPaused:0 ContainersStopped:1 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:53 OomKillDisable:true NGoroutines:71 SystemTime:2024-03-05 04:34:20.897965843 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1055-aws OperatingSystem:Ubuntu 20.04.6 LTS OSType:linux Architecture:aar
ch64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:2 MemTotal:8215040000 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ip-172-31-30-239 Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=builtin] ProductLicense: Warnings:<nil> ServerErro
rs:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.6]] Warnings:<nil>}}
I0305 04:34:20.913034 2604473 start_flags.go:947] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0305 04:34:20.913114 2604473 cni.go:84] Creating CNI manager for ""
I0305 04:34:20.913128 2604473 cni.go:143] "docker" driver + "containerd" runtime found, recommending kindnet
I0305 04:34:20.913177 2604473 start.go:340] cluster config:
{Name:old-k8s-version-929157 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.0 ClusterName:old-k8s-version-929157 Namespace:default APIServerHAVIP: APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local
ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.20.0 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[dashboard:true default-storageclass:true metrics-server:true storage-provisioner:true] CustomAddonImages:map[MetricsScraper:registry.k8s.io/echoserver:1.4 MetricsServer:registry.k8s.io/echoserver:1.4] CustomAddonRegistries:map[MetricsServer:fake.domain] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:do
cker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0305 04:34:20.916895 2604473 out.go:177] * Starting "old-k8s-version-929157" primary control-plane node in "old-k8s-version-929157" cluster
I0305 04:34:20.918964 2604473 cache.go:121] Beginning downloading kic base image for docker with containerd
I0305 04:34:20.921265 2604473 out.go:177] * Pulling base image v0.0.42-1708944392-18244 ...
I0305 04:34:20.923198 2604473 preload.go:132] Checking if preload exists for k8s version v1.20.0 and runtime containerd
I0305 04:34:20.923255 2604473 preload.go:147] Found local preload: /home/jenkins/minikube-integration/17909-2401808/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.20.0-containerd-overlay2-arm64.tar.lz4
I0305 04:34:20.923269 2604473 cache.go:56] Caching tarball of preloaded images
I0305 04:34:20.923368 2604473 preload.go:173] Found /home/jenkins/minikube-integration/17909-2401808/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.20.0-containerd-overlay2-arm64.tar.lz4 in cache, skipping download
I0305 04:34:20.923383 2604473 cache.go:59] Finished verifying existence of preloaded tar for v1.20.0 on containerd
I0305 04:34:20.923435 2604473 image.go:79] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 in local docker daemon
I0305 04:34:20.923728 2604473 profile.go:142] Saving config to /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/old-k8s-version-929157/config.json ...
I0305 04:34:20.947827 2604473 image.go:83] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 in local docker daemon, skipping pull
I0305 04:34:20.947854 2604473 cache.go:144] gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 exists in daemon, skipping load
I0305 04:34:20.947873 2604473 cache.go:194] Successfully downloaded all kic artifacts
I0305 04:34:20.947902 2604473 start.go:360] acquireMachinesLock for old-k8s-version-929157: {Name:mk3ebb374c0860acad12eb74e046b22a8fb5bd23 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0305 04:34:20.947984 2604473 start.go:364] duration metric: took 42.649µs to acquireMachinesLock for "old-k8s-version-929157"
I0305 04:34:20.948004 2604473 start.go:96] Skipping create...Using existing machine configuration
I0305 04:34:20.948009 2604473 fix.go:54] fixHost starting:
I0305 04:34:20.948307 2604473 cli_runner.go:164] Run: docker container inspect old-k8s-version-929157 --format={{.State.Status}}
I0305 04:34:20.964448 2604473 fix.go:112] recreateIfNeeded on old-k8s-version-929157: state=Stopped err=<nil>
W0305 04:34:20.964477 2604473 fix.go:138] unexpected machine state, will restart: <nil>
I0305 04:34:20.966623 2604473 out.go:177] * Restarting existing docker container for "old-k8s-version-929157" ...
I0305 04:34:20.970428 2604473 cli_runner.go:164] Run: docker start old-k8s-version-929157
I0305 04:34:21.438463 2604473 cli_runner.go:164] Run: docker container inspect old-k8s-version-929157 --format={{.State.Status}}
I0305 04:34:21.460724 2604473 kic.go:430] container "old-k8s-version-929157" state is running.
I0305 04:34:21.461125 2604473 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-929157
I0305 04:34:21.501008 2604473 profile.go:142] Saving config to /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/old-k8s-version-929157/config.json ...
I0305 04:34:21.501240 2604473 machine.go:94] provisionDockerMachine start ...
I0305 04:34:21.501302 2604473 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-929157
I0305 04:34:21.527115 2604473 main.go:141] libmachine: Using SSH client type: native
I0305 04:34:21.527396 2604473 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3e1c40] 0x3e44a0 <nil> [] 0s} 127.0.0.1 35352 <nil> <nil>}
I0305 04:34:21.527413 2604473 main.go:141] libmachine: About to run SSH command:
hostname
I0305 04:34:21.528077 2604473 main.go:141] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:56518->127.0.0.1:35352: read: connection reset by peer
I0305 04:34:24.678964 2604473 main.go:141] libmachine: SSH cmd err, output: <nil>: old-k8s-version-929157
I0305 04:34:24.679004 2604473 ubuntu.go:169] provisioning hostname "old-k8s-version-929157"
I0305 04:34:24.679095 2604473 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-929157
I0305 04:34:24.726449 2604473 main.go:141] libmachine: Using SSH client type: native
I0305 04:34:24.726703 2604473 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3e1c40] 0x3e44a0 <nil> [] 0s} 127.0.0.1 35352 <nil> <nil>}
I0305 04:34:24.726722 2604473 main.go:141] libmachine: About to run SSH command:
sudo hostname old-k8s-version-929157 && echo "old-k8s-version-929157" | sudo tee /etc/hostname
I0305 04:34:24.897444 2604473 main.go:141] libmachine: SSH cmd err, output: <nil>: old-k8s-version-929157
I0305 04:34:24.897544 2604473 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-929157
I0305 04:34:24.935682 2604473 main.go:141] libmachine: Using SSH client type: native
I0305 04:34:24.935943 2604473 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3e1c40] 0x3e44a0 <nil> [] 0s} 127.0.0.1 35352 <nil> <nil>}
I0305 04:34:24.935967 2604473 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sold-k8s-version-929157' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 old-k8s-version-929157/g' /etc/hosts;
else
echo '127.0.1.1 old-k8s-version-929157' | sudo tee -a /etc/hosts;
fi
fi
I0305 04:34:25.096175 2604473 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0305 04:34:25.096219 2604473 ubuntu.go:175] set auth options {CertDir:/home/jenkins/minikube-integration/17909-2401808/.minikube CaCertPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca.pem CaPrivateKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/server.pem ServerKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/server-key.pem ClientKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/home/jenkins/minikube-integration/17909-2401808/.minikube}
I0305 04:34:25.096241 2604473 ubuntu.go:177] setting up certificates
I0305 04:34:25.096251 2604473 provision.go:84] configureAuth start
I0305 04:34:25.096325 2604473 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-929157
I0305 04:34:25.159565 2604473 provision.go:143] copyHostCerts
I0305 04:34:25.159641 2604473 exec_runner.go:144] found /home/jenkins/minikube-integration/17909-2401808/.minikube/ca.pem, removing ...
I0305 04:34:25.159657 2604473 exec_runner.go:203] rm: /home/jenkins/minikube-integration/17909-2401808/.minikube/ca.pem
I0305 04:34:25.159740 2604473 exec_runner.go:151] cp: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca.pem --> /home/jenkins/minikube-integration/17909-2401808/.minikube/ca.pem (1082 bytes)
I0305 04:34:25.159852 2604473 exec_runner.go:144] found /home/jenkins/minikube-integration/17909-2401808/.minikube/cert.pem, removing ...
I0305 04:34:25.159863 2604473 exec_runner.go:203] rm: /home/jenkins/minikube-integration/17909-2401808/.minikube/cert.pem
I0305 04:34:25.159891 2604473 exec_runner.go:151] cp: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/cert.pem --> /home/jenkins/minikube-integration/17909-2401808/.minikube/cert.pem (1123 bytes)
I0305 04:34:25.159956 2604473 exec_runner.go:144] found /home/jenkins/minikube-integration/17909-2401808/.minikube/key.pem, removing ...
I0305 04:34:25.159966 2604473 exec_runner.go:203] rm: /home/jenkins/minikube-integration/17909-2401808/.minikube/key.pem
I0305 04:34:25.159992 2604473 exec_runner.go:151] cp: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/key.pem --> /home/jenkins/minikube-integration/17909-2401808/.minikube/key.pem (1679 bytes)
I0305 04:34:25.160086 2604473 provision.go:117] generating server cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/machines/server.pem ca-key=/home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca.pem private-key=/home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca-key.pem org=jenkins.old-k8s-version-929157 san=[127.0.0.1 192.168.76.2 localhost minikube old-k8s-version-929157]
I0305 04:34:25.796613 2604473 provision.go:177] copyRemoteCerts
I0305 04:34:25.796690 2604473 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0305 04:34:25.796733 2604473 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-929157
I0305 04:34:25.818565 2604473 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:35352 SSHKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/old-k8s-version-929157/id_rsa Username:docker}
I0305 04:34:25.913153 2604473 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1082 bytes)
I0305 04:34:25.938951 2604473 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/machines/server.pem --> /etc/docker/server.pem (1233 bytes)
I0305 04:34:25.964093 2604473 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1679 bytes)
I0305 04:34:25.989322 2604473 provision.go:87] duration metric: took 893.047954ms to configureAuth
I0305 04:34:25.989348 2604473 ubuntu.go:193] setting minikube options for container-runtime
I0305 04:34:25.989557 2604473 config.go:182] Loaded profile config "old-k8s-version-929157": Driver=docker, ContainerRuntime=containerd, KubernetesVersion=v1.20.0
I0305 04:34:25.989572 2604473 machine.go:97] duration metric: took 4.488323134s to provisionDockerMachine
I0305 04:34:25.989579 2604473 start.go:293] postStartSetup for "old-k8s-version-929157" (driver="docker")
I0305 04:34:25.989590 2604473 start.go:322] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0305 04:34:25.989640 2604473 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0305 04:34:25.989687 2604473 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-929157
I0305 04:34:26.010627 2604473 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:35352 SSHKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/old-k8s-version-929157/id_rsa Username:docker}
I0305 04:34:26.112878 2604473 ssh_runner.go:195] Run: cat /etc/os-release
I0305 04:34:26.116287 2604473 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0305 04:34:26.116326 2604473 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0305 04:34:26.116339 2604473 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0305 04:34:26.116346 2604473 info.go:137] Remote host: Ubuntu 22.04.3 LTS
I0305 04:34:26.116356 2604473 filesync.go:126] Scanning /home/jenkins/minikube-integration/17909-2401808/.minikube/addons for local assets ...
I0305 04:34:26.116426 2604473 filesync.go:126] Scanning /home/jenkins/minikube-integration/17909-2401808/.minikube/files for local assets ...
I0305 04:34:26.116512 2604473 filesync.go:149] local asset: /home/jenkins/minikube-integration/17909-2401808/.minikube/files/etc/ssl/certs/24071502.pem -> 24071502.pem in /etc/ssl/certs
I0305 04:34:26.116616 2604473 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0305 04:34:26.125617 2604473 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/files/etc/ssl/certs/24071502.pem --> /etc/ssl/certs/24071502.pem (1708 bytes)
I0305 04:34:26.151308 2604473 start.go:296] duration metric: took 161.713656ms for postStartSetup
I0305 04:34:26.151389 2604473 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0305 04:34:26.151474 2604473 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-929157
I0305 04:34:26.167246 2604473 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:35352 SSHKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/old-k8s-version-929157/id_rsa Username:docker}
I0305 04:34:26.257848 2604473 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0305 04:34:26.264310 2604473 fix.go:56] duration metric: took 5.316294068s for fixHost
I0305 04:34:26.264345 2604473 start.go:83] releasing machines lock for "old-k8s-version-929157", held for 5.316353218s
I0305 04:34:26.264432 2604473 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-929157
I0305 04:34:26.283488 2604473 ssh_runner.go:195] Run: cat /version.json
I0305 04:34:26.283546 2604473 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-929157
I0305 04:34:26.283824 2604473 ssh_runner.go:195] Run: curl -sS -m 2 https://registry.k8s.io/
I0305 04:34:26.283896 2604473 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-929157
I0305 04:34:26.327890 2604473 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:35352 SSHKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/old-k8s-version-929157/id_rsa Username:docker}
I0305 04:34:26.335400 2604473 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:35352 SSHKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/old-k8s-version-929157/id_rsa Username:docker}
I0305 04:34:26.439126 2604473 ssh_runner.go:195] Run: systemctl --version
I0305 04:34:26.575298 2604473 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
I0305 04:34:26.580302 2604473 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *loopback.conf* -not -name *.mk_disabled -exec sh -c "grep -q loopback {} && ( grep -q name {} || sudo sed -i '/"type": "loopback"/i \ \ \ \ "name": "loopback",' {} ) && sudo sed -i 's|"cniVersion": ".*"|"cniVersion": "1.0.0"|g' {}" ;
I0305 04:34:26.603824 2604473 cni.go:230] loopback cni configuration patched: "/etc/cni/net.d/*loopback.conf*" found
I0305 04:34:26.603953 2604473 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f ( ( -name *bridge* -or -name *podman* ) -and -not -name *.mk_disabled ) -printf "%p, " -exec sh -c "sudo mv {} {}.mk_disabled" ;
I0305 04:34:26.613076 2604473 cni.go:259] no active bridge cni configs found in "/etc/cni/net.d" - nothing to disable
I0305 04:34:26.613152 2604473 start.go:494] detecting cgroup driver to use...
I0305 04:34:26.613197 2604473 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0305 04:34:26.613279 2604473 ssh_runner.go:195] Run: sudo systemctl stop -f crio
I0305 04:34:26.627905 2604473 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0305 04:34:26.640928 2604473 docker.go:217] disabling cri-docker service (if available) ...
I0305 04:34:26.641050 2604473 ssh_runner.go:195] Run: sudo systemctl stop -f cri-docker.socket
I0305 04:34:26.654390 2604473 ssh_runner.go:195] Run: sudo systemctl stop -f cri-docker.service
I0305 04:34:26.669482 2604473 ssh_runner.go:195] Run: sudo systemctl disable cri-docker.socket
I0305 04:34:26.777195 2604473 ssh_runner.go:195] Run: sudo systemctl mask cri-docker.service
I0305 04:34:26.890560 2604473 docker.go:233] disabling docker service ...
I0305 04:34:26.890687 2604473 ssh_runner.go:195] Run: sudo systemctl stop -f docker.socket
I0305 04:34:26.904298 2604473 ssh_runner.go:195] Run: sudo systemctl stop -f docker.service
I0305 04:34:26.916000 2604473 ssh_runner.go:195] Run: sudo systemctl disable docker.socket
I0305 04:34:27.036435 2604473 ssh_runner.go:195] Run: sudo systemctl mask docker.service
I0305 04:34:27.157433 2604473 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service docker
I0305 04:34:27.171219 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0305 04:34:27.191668 2604473 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.2"|' /etc/containerd/config.toml"
I0305 04:34:27.203655 2604473 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I0305 04:34:27.214512 2604473 containerd.go:146] configuring containerd to use "cgroupfs" as cgroup driver...
I0305 04:34:27.214627 2604473 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = false|g' /etc/containerd/config.toml"
I0305 04:34:27.226104 2604473 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0305 04:34:27.237105 2604473 ssh_runner.go:195] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I0305 04:34:27.249573 2604473 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0305 04:34:27.260705 2604473 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0305 04:34:27.272490 2604473 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I0305 04:34:27.286491 2604473 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0305 04:34:27.298082 2604473 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0305 04:34:27.308635 2604473 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0305 04:34:27.432706 2604473 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0305 04:34:27.696804 2604473 start.go:541] Will wait 60s for socket path /run/containerd/containerd.sock
I0305 04:34:27.696898 2604473 ssh_runner.go:195] Run: stat /run/containerd/containerd.sock
I0305 04:34:27.700961 2604473 start.go:562] Will wait 60s for crictl version
I0305 04:34:27.701043 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:34:27.705776 2604473 ssh_runner.go:195] Run: sudo /usr/bin/crictl version
I0305 04:34:27.747417 2604473 start.go:578] Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: 1.6.28
RuntimeApiVersion: v1
I0305 04:34:27.747580 2604473 ssh_runner.go:195] Run: containerd --version
I0305 04:34:27.779863 2604473 ssh_runner.go:195] Run: containerd --version
I0305 04:34:27.806083 2604473 out.go:177] * Preparing Kubernetes v1.20.0 on containerd 1.6.28 ...
I0305 04:34:27.808096 2604473 cli_runner.go:164] Run: docker network inspect old-k8s-version-929157 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0305 04:34:27.824355 2604473 ssh_runner.go:195] Run: grep 192.168.76.1 host.minikube.internal$ /etc/hosts
I0305 04:34:27.832573 2604473 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.76.1 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0305 04:34:27.844954 2604473 kubeadm.go:877] updating cluster {Name:old-k8s-version-929157 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.0 ClusterName:old-k8s-version-929157 Namespace:default APIServerHAVIP: APIServerName:minik
ubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.20.0 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[dashboard:true default-storageclass:true metrics-server:true storage-provisioner:true] CustomAddonImages:map[MetricsScraper:registry.k8s.io/echoserver:1.4 MetricsServer:registry.k8s.io/echoserver:1.4] CustomAddonRegistries:map[MetricsServer:fake.domain] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/
home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s} ...
I0305 04:34:27.845084 2604473 preload.go:132] Checking if preload exists for k8s version v1.20.0 and runtime containerd
I0305 04:34:27.845149 2604473 ssh_runner.go:195] Run: sudo crictl images --output json
I0305 04:34:27.885739 2604473 containerd.go:612] all images are preloaded for containerd runtime.
I0305 04:34:27.885763 2604473 containerd.go:519] Images already preloaded, skipping extraction
I0305 04:34:27.885829 2604473 ssh_runner.go:195] Run: sudo crictl images --output json
I0305 04:34:27.922371 2604473 containerd.go:612] all images are preloaded for containerd runtime.
I0305 04:34:27.922395 2604473 cache_images.go:84] Images are preloaded, skipping loading
I0305 04:34:27.922403 2604473 kubeadm.go:928] updating node { 192.168.76.2 8443 v1.20.0 containerd true true} ...
I0305 04:34:27.922540 2604473 kubeadm.go:940] kubelet [Unit]
Wants=containerd.service
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.20.0/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --hostname-override=old-k8s-version-929157 --kubeconfig=/etc/kubernetes/kubelet.conf --network-plugin=cni --node-ip=192.168.76.2
[Install]
config:
{KubernetesVersion:v1.20.0 ClusterName:old-k8s-version-929157 Namespace:default APIServerHAVIP: APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:}
I0305 04:34:27.922610 2604473 ssh_runner.go:195] Run: sudo crictl info
I0305 04:34:27.961895 2604473 cni.go:84] Creating CNI manager for ""
I0305 04:34:27.961918 2604473 cni.go:143] "docker" driver + "containerd" runtime found, recommending kindnet
I0305 04:34:27.961928 2604473 kubeadm.go:84] Using pod CIDR: 10.244.0.0/16
I0305 04:34:27.961948 2604473 kubeadm.go:181] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.76.2 APIServerPort:8443 KubernetesVersion:v1.20.0 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:old-k8s-version-929157 NodeName:old-k8s-version-929157 DNSDomain:cluster.local CRISocket:/run/containerd/containerd.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.76.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.76.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt
StaticPodPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[hairpinMode:hairpin-veth runtimeRequestTimeout:15m] PrependCriSocketUnix:false}
I0305 04:34:27.962076 2604473 kubeadm.go:187] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.76.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: /run/containerd/containerd.sock
name: "old-k8s-version-929157"
kubeletExtraArgs:
node-ip: 192.168.76.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.76.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: mk
controlPlaneEndpoint: control-plane.minikube.internal:8443
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
proxy-refresh-interval: "70000"
kubernetesVersion: v1.20.0
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0305 04:34:27.962154 2604473 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.20.0
I0305 04:34:27.971203 2604473 binaries.go:44] Found k8s binaries, skipping transfer
I0305 04:34:27.971274 2604473 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0305 04:34:27.980298 2604473 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (442 bytes)
I0305 04:34:28.001499 2604473 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0305 04:34:28.030172 2604473 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2125 bytes)
I0305 04:34:28.050652 2604473 ssh_runner.go:195] Run: grep 192.168.76.2 control-plane.minikube.internal$ /etc/hosts
I0305 04:34:28.054839 2604473 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.76.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0305 04:34:28.066559 2604473 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0305 04:34:28.182781 2604473 ssh_runner.go:195] Run: sudo systemctl start kubelet
I0305 04:34:28.203121 2604473 certs.go:68] Setting up /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/old-k8s-version-929157 for IP: 192.168.76.2
I0305 04:34:28.203147 2604473 certs.go:194] generating shared ca certs ...
I0305 04:34:28.203164 2604473 certs.go:226] acquiring lock for ca certs: {Name:mkcd340b494959143cea31c1bd7ad2cead6bae3a Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0305 04:34:28.203314 2604473 certs.go:235] skipping valid "minikubeCA" ca cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/ca.key
I0305 04:34:28.203371 2604473 certs.go:235] skipping valid "proxyClientCA" ca cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/proxy-client-ca.key
I0305 04:34:28.203382 2604473 certs.go:256] generating profile certs ...
I0305 04:34:28.203492 2604473 certs.go:359] skipping valid signed profile cert regeneration for "minikube-user": /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/old-k8s-version-929157/client.key
I0305 04:34:28.203564 2604473 certs.go:359] skipping valid signed profile cert regeneration for "minikube": /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/old-k8s-version-929157/apiserver.key.29c00512
I0305 04:34:28.203600 2604473 certs.go:359] skipping valid signed profile cert regeneration for "aggregator": /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/old-k8s-version-929157/proxy-client.key
I0305 04:34:28.203733 2604473 certs.go:484] found cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/2407150.pem (1338 bytes)
W0305 04:34:28.203763 2604473 certs.go:480] ignoring /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/2407150_empty.pem, impossibly tiny 0 bytes
I0305 04:34:28.203773 2604473 certs.go:484] found cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca-key.pem (1675 bytes)
I0305 04:34:28.203797 2604473 certs.go:484] found cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca.pem (1082 bytes)
I0305 04:34:28.203821 2604473 certs.go:484] found cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/cert.pem (1123 bytes)
I0305 04:34:28.203842 2604473 certs.go:484] found cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/key.pem (1679 bytes)
I0305 04:34:28.203892 2604473 certs.go:484] found cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/files/etc/ssl/certs/24071502.pem (1708 bytes)
I0305 04:34:28.204605 2604473 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0305 04:34:28.236338 2604473 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1679 bytes)
I0305 04:34:28.266918 2604473 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0305 04:34:28.296679 2604473 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1675 bytes)
I0305 04:34:28.328823 2604473 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/old-k8s-version-929157/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1432 bytes)
I0305 04:34:28.361672 2604473 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/old-k8s-version-929157/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1675 bytes)
I0305 04:34:28.390969 2604473 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/old-k8s-version-929157/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0305 04:34:28.416401 2604473 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/old-k8s-version-929157/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1675 bytes)
I0305 04:34:28.441108 2604473 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0305 04:34:28.469178 2604473 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/2407150.pem --> /usr/share/ca-certificates/2407150.pem (1338 bytes)
I0305 04:34:28.498753 2604473 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/files/etc/ssl/certs/24071502.pem --> /usr/share/ca-certificates/24071502.pem (1708 bytes)
I0305 04:34:28.524489 2604473 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0305 04:34:28.543865 2604473 ssh_runner.go:195] Run: openssl version
I0305 04:34:28.549527 2604473 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0305 04:34:28.559275 2604473 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0305 04:34:28.563038 2604473 certs.go:528] hashing: -rw-r--r-- 1 root root 1111 Mar 5 03:48 /usr/share/ca-certificates/minikubeCA.pem
I0305 04:34:28.563114 2604473 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0305 04:34:28.570141 2604473 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0305 04:34:28.579787 2604473 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/2407150.pem && ln -fs /usr/share/ca-certificates/2407150.pem /etc/ssl/certs/2407150.pem"
I0305 04:34:28.593785 2604473 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/2407150.pem
I0305 04:34:28.600489 2604473 certs.go:528] hashing: -rw-r--r-- 1 root root 1338 Mar 5 03:53 /usr/share/ca-certificates/2407150.pem
I0305 04:34:28.600626 2604473 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/2407150.pem
I0305 04:34:28.609661 2604473 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/2407150.pem /etc/ssl/certs/51391683.0"
I0305 04:34:28.622428 2604473 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/24071502.pem && ln -fs /usr/share/ca-certificates/24071502.pem /etc/ssl/certs/24071502.pem"
I0305 04:34:28.634679 2604473 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/24071502.pem
I0305 04:34:28.639773 2604473 certs.go:528] hashing: -rw-r--r-- 1 root root 1708 Mar 5 03:53 /usr/share/ca-certificates/24071502.pem
I0305 04:34:28.639869 2604473 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/24071502.pem
I0305 04:34:28.649407 2604473 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/24071502.pem /etc/ssl/certs/3ec20f2e.0"
I0305 04:34:28.664246 2604473 ssh_runner.go:195] Run: stat /var/lib/minikube/certs/apiserver-kubelet-client.crt
I0305 04:34:28.668852 2604473 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/apiserver-etcd-client.crt -checkend 86400
I0305 04:34:28.676857 2604473 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/apiserver-kubelet-client.crt -checkend 86400
I0305 04:34:28.684552 2604473 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/etcd/server.crt -checkend 86400
I0305 04:34:28.692564 2604473 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/etcd/healthcheck-client.crt -checkend 86400
I0305 04:34:28.702568 2604473 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/etcd/peer.crt -checkend 86400
I0305 04:34:28.712702 2604473 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/front-proxy-client.crt -checkend 86400
I0305 04:34:28.722458 2604473 kubeadm.go:391] StartCluster: {Name:old-k8s-version-929157 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.0 ClusterName:old-k8s-version-929157 Namespace:default APIServerHAVIP: APIServerName:minikube
CA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.20.0 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[dashboard:true default-storageclass:true metrics-server:true storage-provisioner:true] CustomAddonImages:map[MetricsScraper:registry.k8s.io/echoserver:1.4 MetricsServer:registry.k8s.io/echoserver:1.4] CustomAddonRegistries:map[MetricsServer:fake.domain] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/hom
e/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0305 04:34:28.722574 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:paused Name: Namespaces:[kube-system]}
I0305 04:34:28.722671 2604473 ssh_runner.go:195] Run: sudo -s eval "crictl ps -a --quiet --label io.kubernetes.pod.namespace=kube-system"
I0305 04:34:28.785458 2604473 cri.go:89] found id: "508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99"
I0305 04:34:28.785493 2604473 cri.go:89] found id: "87df52e13afebd954949a5161f78c6cc7a7f6b310e322390e1794bc00f62fff9"
I0305 04:34:28.785498 2604473 cri.go:89] found id: "acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f"
I0305 04:34:28.785503 2604473 cri.go:89] found id: "feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232"
I0305 04:34:28.785506 2604473 cri.go:89] found id: "fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259"
I0305 04:34:28.785518 2604473 cri.go:89] found id: "27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636"
I0305 04:34:28.785522 2604473 cri.go:89] found id: "35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605"
I0305 04:34:28.785555 2604473 cri.go:89] found id: "e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341"
I0305 04:34:28.785567 2604473 cri.go:89] found id: ""
I0305 04:34:28.785647 2604473 ssh_runner.go:195] Run: sudo runc --root /run/containerd/runc/k8s.io list -f json
I0305 04:34:28.802605 2604473 cri.go:116] JSON = null
W0305 04:34:28.802683 2604473 kubeadm.go:398] unpause failed: list paused: list returned 0 containers, but ps returned 8
I0305 04:34:28.802801 2604473 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
W0305 04:34:28.813481 2604473 kubeadm.go:404] apiserver tunnel failed: apiserver port not set
I0305 04:34:28.813512 2604473 kubeadm.go:407] found existing configuration files, will attempt cluster restart
I0305 04:34:28.813518 2604473 kubeadm.go:587] restartPrimaryControlPlane start ...
I0305 04:34:28.813605 2604473 ssh_runner.go:195] Run: sudo test -d /data/minikube
I0305 04:34:28.826654 2604473 kubeadm.go:129] /data/minikube skipping compat symlinks: sudo test -d /data/minikube: Process exited with status 1
stdout:
stderr:
I0305 04:34:28.827188 2604473 kubeconfig.go:47] verify endpoint returned: get endpoint: "old-k8s-version-929157" does not appear in /home/jenkins/minikube-integration/17909-2401808/kubeconfig
I0305 04:34:28.827359 2604473 kubeconfig.go:62] /home/jenkins/minikube-integration/17909-2401808/kubeconfig needs updating (will repair): [kubeconfig missing "old-k8s-version-929157" cluster setting kubeconfig missing "old-k8s-version-929157" context setting]
I0305 04:34:28.827724 2604473 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17909-2401808/kubeconfig: {Name:mke72a8d501356b8578ef0ffa055ce767e4b48a8 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0305 04:34:28.828999 2604473 ssh_runner.go:195] Run: sudo diff -u /var/tmp/minikube/kubeadm.yaml /var/tmp/minikube/kubeadm.yaml.new
I0305 04:34:28.838373 2604473 kubeadm.go:624] The running cluster does not require reconfiguration: 192.168.76.2
I0305 04:34:28.838407 2604473 kubeadm.go:591] duration metric: took 24.883483ms to restartPrimaryControlPlane
I0305 04:34:28.838417 2604473 kubeadm.go:393] duration metric: took 115.967986ms to StartCluster
I0305 04:34:28.838460 2604473 settings.go:142] acquiring lock: {Name:mke2f647a11201d07ca3bb011de766b9c0e76480 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0305 04:34:28.838541 2604473 settings.go:150] Updating kubeconfig: /home/jenkins/minikube-integration/17909-2401808/kubeconfig
I0305 04:34:28.839320 2604473 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17909-2401808/kubeconfig: {Name:mke72a8d501356b8578ef0ffa055ce767e4b48a8 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0305 04:34:28.839608 2604473 start.go:234] Will wait 6m0s for node &{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.20.0 ContainerRuntime:containerd ControlPlane:true Worker:true}
I0305 04:34:28.841886 2604473 out.go:177] * Verifying Kubernetes components...
I0305 04:34:28.839990 2604473 config.go:182] Loaded profile config "old-k8s-version-929157": Driver=docker, ContainerRuntime=containerd, KubernetesVersion=v1.20.0
I0305 04:34:28.840007 2604473 addons.go:502] enable addons start: toEnable=map[ambassador:false auto-pause:false cloud-spanner:false csi-hostpath-driver:false dashboard:true default-storageclass:true efk:false freshpod:false gcp-auth:false gvisor:false headlamp:false helm-tiller:false inaccel:false ingress:false ingress-dns:false inspektor-gadget:false istio:false istio-provisioner:false kong:false kubeflow:false kubevirt:false logviewer:false metallb:false metrics-server:true nvidia-device-plugin:false nvidia-driver-installer:false nvidia-gpu-device-plugin:false olm:false pod-security-policy:false portainer:false registry:false registry-aliases:false registry-creds:false storage-provisioner:true storage-provisioner-gluster:false storage-provisioner-rancher:false volumesnapshots:false yakd:false]
I0305 04:34:28.843814 2604473 addons.go:69] Setting storage-provisioner=true in profile "old-k8s-version-929157"
I0305 04:34:28.843821 2604473 addons.go:69] Setting dashboard=true in profile "old-k8s-version-929157"
I0305 04:34:28.843855 2604473 addons.go:234] Setting addon storage-provisioner=true in "old-k8s-version-929157"
I0305 04:34:28.843858 2604473 addons.go:234] Setting addon dashboard=true in "old-k8s-version-929157"
W0305 04:34:28.843864 2604473 addons.go:243] addon storage-provisioner should already be in state true
W0305 04:34:28.843895 2604473 addons.go:243] addon dashboard should already be in state true
I0305 04:34:28.843900 2604473 host.go:66] Checking if "old-k8s-version-929157" exists ...
I0305 04:34:28.843926 2604473 host.go:66] Checking if "old-k8s-version-929157" exists ...
I0305 04:34:28.844420 2604473 cli_runner.go:164] Run: docker container inspect old-k8s-version-929157 --format={{.State.Status}}
I0305 04:34:28.844449 2604473 addons.go:69] Setting metrics-server=true in profile "old-k8s-version-929157"
I0305 04:34:28.844472 2604473 addons.go:234] Setting addon metrics-server=true in "old-k8s-version-929157"
W0305 04:34:28.844478 2604473 addons.go:243] addon metrics-server should already be in state true
I0305 04:34:28.844508 2604473 host.go:66] Checking if "old-k8s-version-929157" exists ...
I0305 04:34:28.845025 2604473 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0305 04:34:28.844423 2604473 cli_runner.go:164] Run: docker container inspect old-k8s-version-929157 --format={{.State.Status}}
I0305 04:34:28.845379 2604473 cli_runner.go:164] Run: docker container inspect old-k8s-version-929157 --format={{.State.Status}}
I0305 04:34:28.844434 2604473 addons.go:69] Setting default-storageclass=true in profile "old-k8s-version-929157"
I0305 04:34:28.845734 2604473 addons_storage_classes.go:33] enableOrDisableStorageClasses default-storageclass=true on "old-k8s-version-929157"
I0305 04:34:28.846062 2604473 cli_runner.go:164] Run: docker container inspect old-k8s-version-929157 --format={{.State.Status}}
I0305 04:34:28.901721 2604473 out.go:177] - Using image fake.domain/registry.k8s.io/echoserver:1.4
I0305 04:34:28.904102 2604473 addons.go:426] installing /etc/kubernetes/addons/metrics-apiservice.yaml
I0305 04:34:28.904130 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/metrics-apiservice.yaml (424 bytes)
I0305 04:34:28.904193 2604473 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-929157
I0305 04:34:28.934117 2604473 out.go:177] - Using image docker.io/kubernetesui/dashboard:v2.7.0
I0305 04:34:28.936449 2604473 out.go:177] - Using image registry.k8s.io/echoserver:1.4
I0305 04:34:28.937920 2604473 addons.go:426] installing /etc/kubernetes/addons/dashboard-ns.yaml
I0305 04:34:28.937941 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/dashboard-ns.yaml (759 bytes)
I0305 04:34:28.938018 2604473 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-929157
I0305 04:34:28.947094 2604473 out.go:177] - Using image gcr.io/k8s-minikube/storage-provisioner:v5
I0305 04:34:28.950743 2604473 addons.go:426] installing /etc/kubernetes/addons/storage-provisioner.yaml
I0305 04:34:28.950772 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/storage-provisioner.yaml (2676 bytes)
I0305 04:34:28.950850 2604473 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-929157
I0305 04:34:28.952121 2604473 addons.go:234] Setting addon default-storageclass=true in "old-k8s-version-929157"
W0305 04:34:28.952149 2604473 addons.go:243] addon default-storageclass should already be in state true
I0305 04:34:28.952174 2604473 host.go:66] Checking if "old-k8s-version-929157" exists ...
I0305 04:34:28.952590 2604473 cli_runner.go:164] Run: docker container inspect old-k8s-version-929157 --format={{.State.Status}}
I0305 04:34:29.002518 2604473 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:35352 SSHKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/old-k8s-version-929157/id_rsa Username:docker}
I0305 04:34:29.023182 2604473 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:35352 SSHKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/old-k8s-version-929157/id_rsa Username:docker}
I0305 04:34:29.043541 2604473 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:35352 SSHKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/old-k8s-version-929157/id_rsa Username:docker}
I0305 04:34:29.045246 2604473 addons.go:426] installing /etc/kubernetes/addons/storageclass.yaml
I0305 04:34:29.045275 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/storageclass.yaml (271 bytes)
I0305 04:34:29.045340 2604473 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-929157
I0305 04:34:29.074205 2604473 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:35352 SSHKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/old-k8s-version-929157/id_rsa Username:docker}
I0305 04:34:29.126558 2604473 ssh_runner.go:195] Run: sudo systemctl start kubelet
I0305 04:34:29.202440 2604473 node_ready.go:35] waiting up to 6m0s for node "old-k8s-version-929157" to be "Ready" ...
I0305 04:34:29.273289 2604473 addons.go:426] installing /etc/kubernetes/addons/metrics-server-deployment.yaml
I0305 04:34:29.273313 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/metrics-server-deployment.yaml (1825 bytes)
I0305 04:34:29.280734 2604473 addons.go:426] installing /etc/kubernetes/addons/dashboard-clusterrole.yaml
I0305 04:34:29.280760 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/dashboard-clusterrole.yaml (1001 bytes)
I0305 04:34:29.322095 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/storageclass.yaml
I0305 04:34:29.363175 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/storage-provisioner.yaml
I0305 04:34:29.373855 2604473 addons.go:426] installing /etc/kubernetes/addons/metrics-server-rbac.yaml
I0305 04:34:29.373881 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/metrics-server-rbac.yaml (2175 bytes)
I0305 04:34:29.385482 2604473 addons.go:426] installing /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml
I0305 04:34:29.385516 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml (1018 bytes)
I0305 04:34:29.467533 2604473 addons.go:426] installing /etc/kubernetes/addons/metrics-server-service.yaml
I0305 04:34:29.467562 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/metrics-server-service.yaml (446 bytes)
I0305 04:34:29.486690 2604473 addons.go:426] installing /etc/kubernetes/addons/dashboard-configmap.yaml
I0305 04:34:29.486716 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/dashboard-configmap.yaml (837 bytes)
I0305 04:34:29.538716 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
I0305 04:34:29.635371 2604473 addons.go:426] installing /etc/kubernetes/addons/dashboard-dp.yaml
I0305 04:34:29.635400 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/dashboard-dp.yaml (4201 bytes)
W0305 04:34:29.756418 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:29.756453 2604473 retry.go:31] will retry after 235.96023ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
W0305 04:34:29.764358 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:29.764391 2604473 retry.go:31] will retry after 304.063601ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:29.780372 2604473 addons.go:426] installing /etc/kubernetes/addons/dashboard-role.yaml
I0305 04:34:29.780401 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/dashboard-role.yaml (1724 bytes)
W0305 04:34:29.786711 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:29.786745 2604473 retry.go:31] will retry after 203.759223ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:29.814617 2604473 addons.go:426] installing /etc/kubernetes/addons/dashboard-rolebinding.yaml
I0305 04:34:29.814643 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/dashboard-rolebinding.yaml (1046 bytes)
I0305 04:34:29.837070 2604473 addons.go:426] installing /etc/kubernetes/addons/dashboard-sa.yaml
I0305 04:34:29.837101 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/dashboard-sa.yaml (837 bytes)
I0305 04:34:29.859566 2604473 addons.go:426] installing /etc/kubernetes/addons/dashboard-secret.yaml
I0305 04:34:29.859604 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/dashboard-secret.yaml (1389 bytes)
I0305 04:34:29.884973 2604473 addons.go:426] installing /etc/kubernetes/addons/dashboard-svc.yaml
I0305 04:34:29.885003 2604473 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/dashboard-svc.yaml (1294 bytes)
I0305 04:34:29.912999 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
I0305 04:34:29.991319 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
I0305 04:34:29.993571 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml
W0305 04:34:30.048854 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:30.048894 2604473 retry.go:31] will retry after 228.418763ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:30.069215 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml
W0305 04:34:30.273073 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:30.273107 2604473 retry.go:31] will retry after 435.022207ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:30.278515 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
W0305 04:34:30.280025 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:30.280058 2604473 retry.go:31] will retry after 311.258777ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
W0305 04:34:30.331979 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:30.332014 2604473 retry.go:31] will retry after 220.569779ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
W0305 04:34:30.424476 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:30.424516 2604473 retry.go:31] will retry after 554.405678ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:30.552995 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml
I0305 04:34:30.591486 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml
W0305 04:34:30.655264 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:30.655352 2604473 retry.go:31] will retry after 417.399523ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:30.709080 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
W0305 04:34:30.715567 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:30.715662 2604473 retry.go:31] will retry after 697.058994ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
W0305 04:34:30.782170 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:30.782210 2604473 retry.go:31] will retry after 573.166243ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:30.979578 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
W0305 04:34:31.050653 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:31.050739 2604473 retry.go:31] will retry after 466.172515ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:31.072915 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml
W0305 04:34:31.150046 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:31.150109 2604473 retry.go:31] will retry after 718.525197ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:31.203686 2604473 node_ready.go:53] error getting node "old-k8s-version-929157": Get "https://192.168.76.2:8443/api/v1/nodes/old-k8s-version-929157": dial tcp 192.168.76.2:8443: connect: connection refused
I0305 04:34:31.356022 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
I0305 04:34:31.413429 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml
W0305 04:34:31.455757 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:31.455839 2604473 retry.go:31] will retry after 1.01002364s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:31.517117 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
W0305 04:34:31.535328 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:31.535412 2604473 retry.go:31] will retry after 981.226408ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
W0305 04:34:31.606163 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:31.606198 2604473 retry.go:31] will retry after 678.165716ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:31.869344 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml
W0305 04:34:31.963239 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:31.963328 2604473 retry.go:31] will retry after 1.629015855s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:32.285399 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
W0305 04:34:32.360872 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:32.360915 2604473 retry.go:31] will retry after 1.215163974s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:32.466105 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
I0305 04:34:32.517552 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml
W0305 04:34:32.551129 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:32.551163 2604473 retry.go:31] will retry after 815.092389ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
W0305 04:34:32.595538 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:32.595572 2604473 retry.go:31] will retry after 1.20327092s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:33.367384 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
W0305 04:34:33.500962 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:33.500999 2604473 retry.go:31] will retry after 2.078771819s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:33.576334 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
I0305 04:34:33.592781 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml
W0305 04:34:33.681864 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:33.681904 2604473 retry.go:31] will retry after 1.514500227s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
W0305 04:34:33.695674 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:33.695712 2604473 retry.go:31] will retry after 1.455714833s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:33.703329 2604473 node_ready.go:53] error getting node "old-k8s-version-929157": Get "https://192.168.76.2:8443/api/v1/nodes/old-k8s-version-929157": dial tcp 192.168.76.2:8443: connect: connection refused
I0305 04:34:33.799583 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml
W0305 04:34:33.871127 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:33.871159 2604473 retry.go:31] will retry after 1.90729956s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:35.152276 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml
I0305 04:34:35.196838 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
W0305 04:34:35.264434 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:35.264471 2604473 retry.go:31] will retry after 3.098366168s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
W0305 04:34:35.300303 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:35.300340 2604473 retry.go:31] will retry after 3.540610217s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:35.580743 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
W0305 04:34:35.660493 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:35.660532 2604473 retry.go:31] will retry after 1.698879745s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:35.779249 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml
W0305 04:34:35.859436 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:35.859533 2604473 retry.go:31] will retry after 2.027072844s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:36.203586 2604473 node_ready.go:53] error getting node "old-k8s-version-929157": Get "https://192.168.76.2:8443/api/v1/nodes/old-k8s-version-929157": dial tcp 192.168.76.2:8443: connect: connection refused
I0305 04:34:37.360295 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
W0305 04:34:37.611616 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:37.611653 2604473 retry.go:31] will retry after 2.918650106s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0305 04:34:37.887614 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml
I0305 04:34:38.363185 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml
I0305 04:34:38.841155 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
I0305 04:34:40.531012 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
I0305 04:34:48.203858 2604473 node_ready.go:53] error getting node "old-k8s-version-929157": Get "https://192.168.76.2:8443/api/v1/nodes/old-k8s-version-929157": net/http: TLS handshake timeout
I0305 04:34:48.208758 2604473 ssh_runner.go:235] Completed: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: (10.321101527s)
W0305 04:34:48.208790 2604473 addons.go:452] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
Unable to connect to the server: net/http: TLS handshake timeout
I0305 04:34:48.208808 2604473 retry.go:31] will retry after 4.601306653s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
Unable to connect to the server: net/http: TLS handshake timeout
I0305 04:34:48.647146 2604473 node_ready.go:49] node "old-k8s-version-929157" has status "Ready":"True"
I0305 04:34:48.647172 2604473 node_ready.go:38] duration metric: took 19.444688555s for node "old-k8s-version-929157" to be "Ready" ...
I0305 04:34:48.647183 2604473 pod_ready.go:35] extra waiting up to 6m0s for all system-critical pods including labels [k8s-app=kube-dns component=etcd component=kube-apiserver component=kube-controller-manager k8s-app=kube-proxy component=kube-scheduler] to be "Ready" ...
I0305 04:34:48.793846 2604473 pod_ready.go:78] waiting up to 6m0s for pod "coredns-74ff55c5b-k5sj5" in "kube-system" namespace to be "Ready" ...
I0305 04:34:48.897028 2604473 pod_ready.go:92] pod "coredns-74ff55c5b-k5sj5" in "kube-system" namespace has status "Ready":"True"
I0305 04:34:48.897053 2604473 pod_ready.go:81] duration metric: took 103.130912ms for pod "coredns-74ff55c5b-k5sj5" in "kube-system" namespace to be "Ready" ...
I0305 04:34:48.897064 2604473 pod_ready.go:78] waiting up to 6m0s for pod "etcd-old-k8s-version-929157" in "kube-system" namespace to be "Ready" ...
I0305 04:34:48.910122 2604473 pod_ready.go:92] pod "etcd-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"True"
I0305 04:34:48.910182 2604473 pod_ready.go:81] duration metric: took 13.084306ms for pod "etcd-old-k8s-version-929157" in "kube-system" namespace to be "Ready" ...
I0305 04:34:48.910198 2604473 pod_ready.go:78] waiting up to 6m0s for pod "kube-apiserver-old-k8s-version-929157" in "kube-system" namespace to be "Ready" ...
I0305 04:34:48.937569 2604473 pod_ready.go:92] pod "kube-apiserver-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"True"
I0305 04:34:48.937602 2604473 pod_ready.go:81] duration metric: took 27.385823ms for pod "kube-apiserver-old-k8s-version-929157" in "kube-system" namespace to be "Ready" ...
I0305 04:34:48.937616 2604473 pod_ready.go:78] waiting up to 6m0s for pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace to be "Ready" ...
I0305 04:34:49.470448 2604473 ssh_runner.go:235] Completed: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: (11.107222301s)
I0305 04:34:49.761519 2604473 ssh_runner.go:235] Completed: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: (10.920307684s)
I0305 04:34:49.764264 2604473 out.go:177] * Some dashboard features require the metrics-server addon. To enable all features please run:
minikube -p old-k8s-version-929157 addons enable metrics-server
I0305 04:34:49.761811 2604473 ssh_runner.go:235] Completed: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: (9.230737856s)
I0305 04:34:49.764362 2604473 addons.go:470] Verifying addon metrics-server=true in "old-k8s-version-929157"
I0305 04:34:50.944503 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:34:52.810743 2604473 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml
I0305 04:34:52.944780 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:34:53.180308 2604473 out.go:177] * Enabled addons: storage-provisioner, dashboard, metrics-server, default-storageclass
I0305 04:34:53.182341 2604473 addons.go:505] duration metric: took 24.342329689s for enable addons: enabled=[storage-provisioner dashboard metrics-server default-storageclass]
I0305 04:34:55.445526 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:34:57.449257 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:34:59.944212 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:02.444522 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:04.944563 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:06.947279 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:08.947545 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:11.444576 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:13.445398 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:15.447583 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:17.943172 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:19.943749 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:21.943843 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:23.944325 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:25.944493 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:28.443808 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:30.444230 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:32.444396 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:34.444683 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:36.445475 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:38.943938 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:40.945497 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:43.444746 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:45.944428 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:47.945285 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:50.445262 2604473 pod_ready.go:102] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:52.944132 2604473 pod_ready.go:92] pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"True"
I0305 04:35:52.944159 2604473 pod_ready.go:81] duration metric: took 1m4.006535622s for pod "kube-controller-manager-old-k8s-version-929157" in "kube-system" namespace to be "Ready" ...
I0305 04:35:52.944171 2604473 pod_ready.go:78] waiting up to 6m0s for pod "kube-proxy-dxjrs" in "kube-system" namespace to be "Ready" ...
I0305 04:35:52.949976 2604473 pod_ready.go:92] pod "kube-proxy-dxjrs" in "kube-system" namespace has status "Ready":"True"
I0305 04:35:52.949997 2604473 pod_ready.go:81] duration metric: took 5.8183ms for pod "kube-proxy-dxjrs" in "kube-system" namespace to be "Ready" ...
I0305 04:35:52.950013 2604473 pod_ready.go:78] waiting up to 6m0s for pod "kube-scheduler-old-k8s-version-929157" in "kube-system" namespace to be "Ready" ...
I0305 04:35:54.957556 2604473 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:57.457071 2604473 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:35:59.956034 2604473 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:01.956073 2604473 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:03.956140 2604473 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:04.955705 2604473 pod_ready.go:92] pod "kube-scheduler-old-k8s-version-929157" in "kube-system" namespace has status "Ready":"True"
I0305 04:36:04.955730 2604473 pod_ready.go:81] duration metric: took 12.005709575s for pod "kube-scheduler-old-k8s-version-929157" in "kube-system" namespace to be "Ready" ...
I0305 04:36:04.955762 2604473 pod_ready.go:78] waiting up to 6m0s for pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace to be "Ready" ...
I0305 04:36:06.961231 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:08.962543 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:11.462335 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:13.464051 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:15.961997 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:17.962115 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:20.461961 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:22.463335 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:24.463534 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:26.464123 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:28.961798 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:30.962011 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:32.962121 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:34.962183 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:37.462881 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:39.516939 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:41.962088 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:44.463118 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:46.962707 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:49.462004 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:51.462243 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:53.462855 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:55.462944 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:36:57.962321 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:00.462750 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:02.462845 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:04.961437 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:07.461956 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:09.462754 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:11.968628 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:14.462080 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:16.962102 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:18.962262 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:20.962305 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:23.462575 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:25.961857 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:27.962745 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:30.461779 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:32.962057 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:34.962154 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:36.968405 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:39.461937 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:41.461986 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:43.462803 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:45.961418 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:47.962528 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:50.462018 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:52.963607 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:55.461898 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:57.463001 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:37:59.962296 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:02.462348 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:04.462525 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:06.962089 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:09.462096 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:11.462312 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:13.963914 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:16.462494 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:18.961955 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:20.962628 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:22.962996 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:25.461663 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:27.462197 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:29.462578 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:31.961841 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:33.962335 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:36.462338 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:38.473465 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:40.961961 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:43.463273 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:45.961532 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:47.962692 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:50.461234 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:52.962176 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:55.462083 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:57.462783 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:38:59.962235 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:01.962282 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:03.962572 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:06.469061 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:08.961445 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:11.011409 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:13.462842 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:15.961380 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:17.961742 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:19.961966 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:21.962591 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:24.461811 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:26.962297 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:28.962831 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:31.461943 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:33.462754 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:35.467573 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:37.962701 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:40.463280 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:42.961942 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:45.462367 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:47.466725 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:49.961686 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:51.962546 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:54.461786 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:56.462266 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:39:58.462670 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:40:00.493317 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:40:02.962387 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:40:04.966031 2604473 pod_ready.go:102] pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace has status "Ready":"False"
I0305 04:40:04.966062 2604473 pod_ready.go:81] duration metric: took 4m0.010291303s for pod "metrics-server-9975d5f86-wjtl9" in "kube-system" namespace to be "Ready" ...
E0305 04:40:04.966073 2604473 pod_ready.go:66] WaitExtra: waitPodCondition: context deadline exceeded
I0305 04:40:04.966081 2604473 pod_ready.go:38] duration metric: took 5m16.318887355s for extra waiting for all system-critical and pods with labels [k8s-app=kube-dns component=etcd component=kube-apiserver component=kube-controller-manager k8s-app=kube-proxy component=kube-scheduler] to be "Ready" ...
I0305 04:40:04.966097 2604473 api_server.go:52] waiting for apiserver process to appear ...
I0305 04:40:04.966123 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0305 04:40:04.966183 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0305 04:40:05.049153 2604473 cri.go:89] found id: "71cb2e24b42a7f8d797580bf0b66535d6aa909eae6127ef3f669c69251df7f53"
I0305 04:40:05.049176 2604473 cri.go:89] found id: "fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259"
I0305 04:40:05.049181 2604473 cri.go:89] found id: ""
I0305 04:40:05.049189 2604473 logs.go:276] 2 containers: [71cb2e24b42a7f8d797580bf0b66535d6aa909eae6127ef3f669c69251df7f53 fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259]
I0305 04:40:05.049249 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.053628 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.058121 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0305 04:40:05.058193 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0305 04:40:05.127478 2604473 cri.go:89] found id: "3dd18badcf58a90e38fe4db8a253226bf84cf241094215382ff4b9cc24ee04a5"
I0305 04:40:05.127500 2604473 cri.go:89] found id: "35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605"
I0305 04:40:05.127505 2604473 cri.go:89] found id: ""
I0305 04:40:05.127512 2604473 logs.go:276] 2 containers: [3dd18badcf58a90e38fe4db8a253226bf84cf241094215382ff4b9cc24ee04a5 35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605]
I0305 04:40:05.127589 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.133401 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.137630 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0305 04:40:05.137769 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0305 04:40:05.207606 2604473 cri.go:89] found id: "e0dbacaa8e30885944ab46b688f9bbefac308f823788712e6c14ecdc1556f552"
I0305 04:40:05.207682 2604473 cri.go:89] found id: "508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99"
I0305 04:40:05.207700 2604473 cri.go:89] found id: ""
I0305 04:40:05.207724 2604473 logs.go:276] 2 containers: [e0dbacaa8e30885944ab46b688f9bbefac308f823788712e6c14ecdc1556f552 508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99]
I0305 04:40:05.207818 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.212238 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.217228 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0305 04:40:05.217306 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0305 04:40:05.286141 2604473 cri.go:89] found id: "a5d43d0d8a5d3f1e42ca4c1b9ec4dd6a812c6454f3858ec238850f1190bce93c"
I0305 04:40:05.286180 2604473 cri.go:89] found id: "27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636"
I0305 04:40:05.286192 2604473 cri.go:89] found id: ""
I0305 04:40:05.286203 2604473 logs.go:276] 2 containers: [a5d43d0d8a5d3f1e42ca4c1b9ec4dd6a812c6454f3858ec238850f1190bce93c 27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636]
I0305 04:40:05.286275 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.291321 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.296302 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0305 04:40:05.296406 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0305 04:40:05.362301 2604473 cri.go:89] found id: "dd4135e2a8d47ee5af337a3314218c302ed97e8f2c878263df59d7b367ea4443"
I0305 04:40:05.362322 2604473 cri.go:89] found id: "acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f"
I0305 04:40:05.362327 2604473 cri.go:89] found id: ""
I0305 04:40:05.362335 2604473 logs.go:276] 2 containers: [dd4135e2a8d47ee5af337a3314218c302ed97e8f2c878263df59d7b367ea4443 acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f]
I0305 04:40:05.362389 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.366162 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.370100 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0305 04:40:05.370170 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0305 04:40:05.437521 2604473 cri.go:89] found id: "9a1fcc970baf72fae5fc3eb76833e0232c8f51d7938d48ea284de0940c3a961c"
I0305 04:40:05.437541 2604473 cri.go:89] found id: "e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341"
I0305 04:40:05.437550 2604473 cri.go:89] found id: ""
I0305 04:40:05.437594 2604473 logs.go:276] 2 containers: [9a1fcc970baf72fae5fc3eb76833e0232c8f51d7938d48ea284de0940c3a961c e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341]
I0305 04:40:05.437654 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.442547 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.446889 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kindnet Namespaces:[]}
I0305 04:40:05.446970 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kindnet
I0305 04:40:05.502430 2604473 cri.go:89] found id: "767fb3c810f4e121fae9120786476ac6029d586fd1a6203f8cf8917dbbaf13cb"
I0305 04:40:05.502449 2604473 cri.go:89] found id: "feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232"
I0305 04:40:05.502455 2604473 cri.go:89] found id: ""
I0305 04:40:05.502462 2604473 logs.go:276] 2 containers: [767fb3c810f4e121fae9120786476ac6029d586fd1a6203f8cf8917dbbaf13cb feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232]
I0305 04:40:05.502519 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.506974 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.511348 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0305 04:40:05.511414 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0305 04:40:05.564030 2604473 cri.go:89] found id: "4c1bed4d5ba29d0be56313d0a71a5ec6110f707ff84f3f6e34f4028a3d82e073"
I0305 04:40:05.564051 2604473 cri.go:89] found id: ""
I0305 04:40:05.564059 2604473 logs.go:276] 1 containers: [4c1bed4d5ba29d0be56313d0a71a5ec6110f707ff84f3f6e34f4028a3d82e073]
I0305 04:40:05.564117 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.568203 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0305 04:40:05.568282 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0305 04:40:05.626979 2604473 cri.go:89] found id: "9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff"
I0305 04:40:05.627006 2604473 cri.go:89] found id: "cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7"
I0305 04:40:05.627012 2604473 cri.go:89] found id: ""
I0305 04:40:05.627020 2604473 logs.go:276] 2 containers: [9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7]
I0305 04:40:05.627097 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.631684 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.635856 2604473 logs.go:123] Gathering logs for containerd ...
I0305 04:40:05.635888 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0305 04:40:05.707408 2604473 logs.go:123] Gathering logs for kubelet ...
I0305 04:40:05.707485 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0305 04:40:05.768492 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566136 662 reflector.go:138] object-"kube-system"/"metrics-server-token-z55cz": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "metrics-server-token-z55cz" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.768846 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566265 662 reflector.go:138] object-"kube-system"/"kube-proxy-token-w7vc2": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kube-proxy-token-w7vc2" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.769080 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566376 662 reflector.go:138] object-"kube-system"/"coredns": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "coredns" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.769316 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566444 662 reflector.go:138] object-"kube-system"/"coredns-token-dkvsz": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "coredns-token-dkvsz" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.769553 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566500 662 reflector.go:138] object-"default"/"default-token-xt7xb": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "default-token-xt7xb" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "default": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.769802 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566558 662 reflector.go:138] object-"kube-system"/"storage-provisioner-token-hsz7w": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "storage-provisioner-token-hsz7w" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.770040 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566618 662 reflector.go:138] object-"kube-system"/"kindnet-token-jq4vl": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kindnet-token-jq4vl" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.770266 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566677 662 reflector.go:138] object-"kube-system"/"kube-proxy": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "kube-proxy" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.779863 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:51 old-k8s-version-929157 kubelet[662]: E0305 04:34:51.325447 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:05.780103 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:51 old-k8s-version-929157 kubelet[662]: E0305 04:34:51.878905 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.783162 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:05 old-k8s-version-929157 kubelet[662]: E0305 04:35:05.758082 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:05.785363 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:14 old-k8s-version-929157 kubelet[662]: E0305 04:35:14.030339 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.785830 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:15 old-k8s-version-929157 kubelet[662]: E0305 04:35:15.047564 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.786350 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:18 old-k8s-version-929157 kubelet[662]: E0305 04:35:18.750259 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.786679 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:22 old-k8s-version-929157 kubelet[662]: E0305 04:35:22.709995 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.787115 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:23 old-k8s-version-929157 kubelet[662]: E0305 04:35:23.063333 662 pod_workers.go:191] Error syncing pod b3db2690-165e-46e7-bef2-d3020d2c11e2 ("storage-provisioner_kube-system(b3db2690-165e-46e7-bef2-d3020d2c11e2)"), skipping: failed to "StartContainer" for "storage-provisioner" with CrashLoopBackOff: "back-off 10s restarting failed container=storage-provisioner pod=storage-provisioner_kube-system(b3db2690-165e-46e7-bef2-d3020d2c11e2)"
W0305 04:40:05.790026 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:30 old-k8s-version-929157 kubelet[662]: E0305 04:35:30.764359 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:05.790813 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:38 old-k8s-version-929157 kubelet[662]: E0305 04:35:38.133208 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.791171 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:42 old-k8s-version-929157 kubelet[662]: E0305 04:35:42.709684 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.791380 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:44 old-k8s-version-929157 kubelet[662]: E0305 04:35:44.750232 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.791756 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:53 old-k8s-version-929157 kubelet[662]: E0305 04:35:53.749653 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.791968 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:57 old-k8s-version-929157 kubelet[662]: E0305 04:35:57.760380 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.792575 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:08 old-k8s-version-929157 kubelet[662]: E0305 04:36:08.224516 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.792923 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:12 old-k8s-version-929157 kubelet[662]: E0305 04:36:12.710208 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.795355 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:13 old-k8s-version-929157 kubelet[662]: E0305 04:36:13.760058 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:05.795615 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:24 old-k8s-version-929157 kubelet[662]: E0305 04:36:24.750302 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.796035 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:27 old-k8s-version-929157 kubelet[662]: E0305 04:36:27.749613 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.796286 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:35 old-k8s-version-929157 kubelet[662]: E0305 04:36:35.750001 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.796707 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:38 old-k8s-version-929157 kubelet[662]: E0305 04:36:38.749574 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.797944 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:46 old-k8s-version-929157 kubelet[662]: E0305 04:36:46.750380 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.798615 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:51 old-k8s-version-929157 kubelet[662]: E0305 04:36:51.312622 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.798943 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:52 old-k8s-version-929157 kubelet[662]: E0305 04:36:52.709685 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.799128 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:01 old-k8s-version-929157 kubelet[662]: E0305 04:37:01.749989 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.799463 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:06 old-k8s-version-929157 kubelet[662]: E0305 04:37:06.750188 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.799655 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:12 old-k8s-version-929157 kubelet[662]: E0305 04:37:12.752686 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.799981 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:17 old-k8s-version-929157 kubelet[662]: E0305 04:37:17.749715 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.800160 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:25 old-k8s-version-929157 kubelet[662]: E0305 04:37:25.749931 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.800482 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:28 old-k8s-version-929157 kubelet[662]: E0305 04:37:28.749688 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.800809 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:39 old-k8s-version-929157 kubelet[662]: E0305 04:37:39.749886 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.803220 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:39 old-k8s-version-929157 kubelet[662]: E0305 04:37:39.757581 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:05.803602 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:50 old-k8s-version-929157 kubelet[662]: E0305 04:37:50.750942 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.803822 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:50 old-k8s-version-929157 kubelet[662]: E0305 04:37:50.751501 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.804195 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:02 old-k8s-version-929157 kubelet[662]: E0305 04:38:02.750200 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.804439 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:05 old-k8s-version-929157 kubelet[662]: E0305 04:38:05.750061 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.805100 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:18 old-k8s-version-929157 kubelet[662]: E0305 04:38:18.528409 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.805318 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:19 old-k8s-version-929157 kubelet[662]: E0305 04:38:19.750661 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.805692 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:22 old-k8s-version-929157 kubelet[662]: E0305 04:38:22.709687 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.805909 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:33 old-k8s-version-929157 kubelet[662]: E0305 04:38:33.749936 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.806279 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:36 old-k8s-version-929157 kubelet[662]: E0305 04:38:36.756212 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.806486 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:45 old-k8s-version-929157 kubelet[662]: E0305 04:38:45.749980 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.806868 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:48 old-k8s-version-929157 kubelet[662]: E0305 04:38:48.749677 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.807073 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:59 old-k8s-version-929157 kubelet[662]: E0305 04:38:59.750099 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.807464 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:03 old-k8s-version-929157 kubelet[662]: E0305 04:39:03.749594 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.807748 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:11 old-k8s-version-929157 kubelet[662]: E0305 04:39:11.749896 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.808093 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:15 old-k8s-version-929157 kubelet[662]: E0305 04:39:15.749647 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.808322 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:24 old-k8s-version-929157 kubelet[662]: E0305 04:39:24.750162 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.808670 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:26 old-k8s-version-929157 kubelet[662]: E0305 04:39:26.753574 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.808879 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:35 old-k8s-version-929157 kubelet[662]: E0305 04:39:35.750670 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.809232 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:39 old-k8s-version-929157 kubelet[662]: E0305 04:39:39.749986 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.809437 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:46 old-k8s-version-929157 kubelet[662]: E0305 04:39:46.750507 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.809789 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:51 old-k8s-version-929157 kubelet[662]: E0305 04:39:51.749596 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.809997 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:58 old-k8s-version-929157 kubelet[662]: E0305 04:39:58.753750 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.810345 2604473 logs.go:138] Found kubelet problem: Mar 05 04:40:03 old-k8s-version-929157 kubelet[662]: E0305 04:40:03.749667 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
I0305 04:40:05.810369 2604473 logs.go:123] Gathering logs for kube-apiserver [fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259] ...
I0305 04:40:05.810399 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259"
I0305 04:40:05.903991 2604473 logs.go:123] Gathering logs for coredns [e0dbacaa8e30885944ab46b688f9bbefac308f823788712e6c14ecdc1556f552] ...
I0305 04:40:05.904037 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 e0dbacaa8e30885944ab46b688f9bbefac308f823788712e6c14ecdc1556f552"
I0305 04:40:05.973325 2604473 logs.go:123] Gathering logs for kube-scheduler [a5d43d0d8a5d3f1e42ca4c1b9ec4dd6a812c6454f3858ec238850f1190bce93c] ...
I0305 04:40:05.973352 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 a5d43d0d8a5d3f1e42ca4c1b9ec4dd6a812c6454f3858ec238850f1190bce93c"
I0305 04:40:06.039539 2604473 logs.go:123] Gathering logs for kube-scheduler [27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636] ...
I0305 04:40:06.039576 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636"
I0305 04:40:06.101871 2604473 logs.go:123] Gathering logs for kube-controller-manager [e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341] ...
I0305 04:40:06.101965 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341"
I0305 04:40:06.208749 2604473 logs.go:123] Gathering logs for kindnet [767fb3c810f4e121fae9120786476ac6029d586fd1a6203f8cf8917dbbaf13cb] ...
I0305 04:40:06.208798 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 767fb3c810f4e121fae9120786476ac6029d586fd1a6203f8cf8917dbbaf13cb"
I0305 04:40:06.265134 2604473 logs.go:123] Gathering logs for coredns [508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99] ...
I0305 04:40:06.265559 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99"
I0305 04:40:06.326851 2604473 logs.go:123] Gathering logs for kube-proxy [dd4135e2a8d47ee5af337a3314218c302ed97e8f2c878263df59d7b367ea4443] ...
I0305 04:40:06.326882 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 dd4135e2a8d47ee5af337a3314218c302ed97e8f2c878263df59d7b367ea4443"
I0305 04:40:06.402071 2604473 logs.go:123] Gathering logs for kube-proxy [acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f] ...
I0305 04:40:06.402097 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f"
I0305 04:40:06.477887 2604473 logs.go:123] Gathering logs for kube-controller-manager [9a1fcc970baf72fae5fc3eb76833e0232c8f51d7938d48ea284de0940c3a961c] ...
I0305 04:40:06.477930 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 9a1fcc970baf72fae5fc3eb76833e0232c8f51d7938d48ea284de0940c3a961c"
I0305 04:40:06.556386 2604473 logs.go:123] Gathering logs for container status ...
I0305 04:40:06.556423 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0305 04:40:06.640735 2604473 logs.go:123] Gathering logs for dmesg ...
I0305 04:40:06.640819 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0305 04:40:06.660696 2604473 logs.go:123] Gathering logs for describe nodes ...
I0305 04:40:06.660771 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.20.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
I0305 04:40:06.935963 2604473 logs.go:123] Gathering logs for etcd [3dd18badcf58a90e38fe4db8a253226bf84cf241094215382ff4b9cc24ee04a5] ...
I0305 04:40:06.936044 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 3dd18badcf58a90e38fe4db8a253226bf84cf241094215382ff4b9cc24ee04a5"
I0305 04:40:07.014969 2604473 logs.go:123] Gathering logs for etcd [35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605] ...
I0305 04:40:07.017390 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605"
I0305 04:40:07.120671 2604473 logs.go:123] Gathering logs for kindnet [feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232] ...
I0305 04:40:07.120750 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232"
I0305 04:40:07.165436 2604473 logs.go:123] Gathering logs for storage-provisioner [9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff] ...
I0305 04:40:07.165514 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff"
I0305 04:40:07.244226 2604473 logs.go:123] Gathering logs for storage-provisioner [cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7] ...
I0305 04:40:07.244266 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7"
I0305 04:40:07.306233 2604473 logs.go:123] Gathering logs for kube-apiserver [71cb2e24b42a7f8d797580bf0b66535d6aa909eae6127ef3f669c69251df7f53] ...
I0305 04:40:07.306259 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 71cb2e24b42a7f8d797580bf0b66535d6aa909eae6127ef3f669c69251df7f53"
I0305 04:40:07.442298 2604473 logs.go:123] Gathering logs for kubernetes-dashboard [4c1bed4d5ba29d0be56313d0a71a5ec6110f707ff84f3f6e34f4028a3d82e073] ...
I0305 04:40:07.442336 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 4c1bed4d5ba29d0be56313d0a71a5ec6110f707ff84f3f6e34f4028a3d82e073"
I0305 04:40:07.518486 2604473 out.go:304] Setting ErrFile to fd 2...
I0305 04:40:07.518513 2604473 out.go:338] TERM=,COLORTERM=, which probably does not support color
W0305 04:40:07.518562 2604473 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0305 04:40:07.518575 2604473 out.go:239] Mar 05 04:39:39 old-k8s-version-929157 kubelet[662]: E0305 04:39:39.749986 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
Mar 05 04:39:39 old-k8s-version-929157 kubelet[662]: E0305 04:39:39.749986 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:07.518584 2604473 out.go:239] Mar 05 04:39:46 old-k8s-version-929157 kubelet[662]: E0305 04:39:46.750507 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Mar 05 04:39:46 old-k8s-version-929157 kubelet[662]: E0305 04:39:46.750507 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:07.518596 2604473 out.go:239] Mar 05 04:39:51 old-k8s-version-929157 kubelet[662]: E0305 04:39:51.749596 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
Mar 05 04:39:51 old-k8s-version-929157 kubelet[662]: E0305 04:39:51.749596 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:07.518602 2604473 out.go:239] Mar 05 04:39:58 old-k8s-version-929157 kubelet[662]: E0305 04:39:58.753750 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Mar 05 04:39:58 old-k8s-version-929157 kubelet[662]: E0305 04:39:58.753750 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:07.518614 2604473 out.go:239] Mar 05 04:40:03 old-k8s-version-929157 kubelet[662]: E0305 04:40:03.749667 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
Mar 05 04:40:03 old-k8s-version-929157 kubelet[662]: E0305 04:40:03.749667 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
I0305 04:40:07.518622 2604473 out.go:304] Setting ErrFile to fd 2...
I0305 04:40:07.518630 2604473 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0305 04:40:17.520283 2604473 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0305 04:40:17.538481 2604473 api_server.go:72] duration metric: took 5m48.69883193s to wait for apiserver process to appear ...
I0305 04:40:17.538502 2604473 api_server.go:88] waiting for apiserver healthz status ...
I0305 04:40:17.538536 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0305 04:40:17.538590 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0305 04:40:17.587028 2604473 cri.go:89] found id: "71cb2e24b42a7f8d797580bf0b66535d6aa909eae6127ef3f669c69251df7f53"
I0305 04:40:17.587054 2604473 cri.go:89] found id: "fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259"
I0305 04:40:17.587059 2604473 cri.go:89] found id: ""
I0305 04:40:17.587066 2604473 logs.go:276] 2 containers: [71cb2e24b42a7f8d797580bf0b66535d6aa909eae6127ef3f669c69251df7f53 fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259]
I0305 04:40:17.587121 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.590809 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.594269 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0305 04:40:17.594337 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0305 04:40:17.647126 2604473 cri.go:89] found id: "3dd18badcf58a90e38fe4db8a253226bf84cf241094215382ff4b9cc24ee04a5"
I0305 04:40:17.647155 2604473 cri.go:89] found id: "35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605"
I0305 04:40:17.647160 2604473 cri.go:89] found id: ""
I0305 04:40:17.647167 2604473 logs.go:276] 2 containers: [3dd18badcf58a90e38fe4db8a253226bf84cf241094215382ff4b9cc24ee04a5 35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605]
I0305 04:40:17.647267 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.651142 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.654792 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0305 04:40:17.654857 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0305 04:40:17.714047 2604473 cri.go:89] found id: "e0dbacaa8e30885944ab46b688f9bbefac308f823788712e6c14ecdc1556f552"
I0305 04:40:17.714066 2604473 cri.go:89] found id: "508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99"
I0305 04:40:17.714070 2604473 cri.go:89] found id: ""
I0305 04:40:17.714078 2604473 logs.go:276] 2 containers: [e0dbacaa8e30885944ab46b688f9bbefac308f823788712e6c14ecdc1556f552 508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99]
I0305 04:40:17.714132 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.717844 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.721446 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0305 04:40:17.721558 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0305 04:40:17.836275 2604473 cri.go:89] found id: "a5d43d0d8a5d3f1e42ca4c1b9ec4dd6a812c6454f3858ec238850f1190bce93c"
I0305 04:40:17.836346 2604473 cri.go:89] found id: "27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636"
I0305 04:40:17.836368 2604473 cri.go:89] found id: ""
I0305 04:40:17.836390 2604473 logs.go:276] 2 containers: [a5d43d0d8a5d3f1e42ca4c1b9ec4dd6a812c6454f3858ec238850f1190bce93c 27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636]
I0305 04:40:17.836479 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.843604 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.852424 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0305 04:40:17.852530 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0305 04:40:17.959522 2604473 cri.go:89] found id: "dd4135e2a8d47ee5af337a3314218c302ed97e8f2c878263df59d7b367ea4443"
I0305 04:40:17.959555 2604473 cri.go:89] found id: "acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f"
I0305 04:40:17.959560 2604473 cri.go:89] found id: ""
I0305 04:40:17.959567 2604473 logs.go:276] 2 containers: [dd4135e2a8d47ee5af337a3314218c302ed97e8f2c878263df59d7b367ea4443 acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f]
I0305 04:40:17.959671 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.964637 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.969480 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0305 04:40:17.969577 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0305 04:40:18.062276 2604473 cri.go:89] found id: "9a1fcc970baf72fae5fc3eb76833e0232c8f51d7938d48ea284de0940c3a961c"
I0305 04:40:18.062348 2604473 cri.go:89] found id: "e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341"
I0305 04:40:18.062368 2604473 cri.go:89] found id: ""
I0305 04:40:18.062390 2604473 logs.go:276] 2 containers: [9a1fcc970baf72fae5fc3eb76833e0232c8f51d7938d48ea284de0940c3a961c e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341]
I0305 04:40:18.062474 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:18.067015 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:18.071263 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kindnet Namespaces:[]}
I0305 04:40:18.071413 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kindnet
I0305 04:40:18.131770 2604473 cri.go:89] found id: "767fb3c810f4e121fae9120786476ac6029d586fd1a6203f8cf8917dbbaf13cb"
I0305 04:40:18.131845 2604473 cri.go:89] found id: "feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232"
I0305 04:40:18.131867 2604473 cri.go:89] found id: ""
I0305 04:40:18.131892 2604473 logs.go:276] 2 containers: [767fb3c810f4e121fae9120786476ac6029d586fd1a6203f8cf8917dbbaf13cb feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232]
I0305 04:40:18.131974 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:18.136721 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:18.140918 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0305 04:40:18.141043 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0305 04:40:18.192151 2604473 cri.go:89] found id: "4c1bed4d5ba29d0be56313d0a71a5ec6110f707ff84f3f6e34f4028a3d82e073"
I0305 04:40:18.192224 2604473 cri.go:89] found id: ""
I0305 04:40:18.192247 2604473 logs.go:276] 1 containers: [4c1bed4d5ba29d0be56313d0a71a5ec6110f707ff84f3f6e34f4028a3d82e073]
I0305 04:40:18.192329 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:18.197650 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0305 04:40:18.197770 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0305 04:40:18.248230 2604473 cri.go:89] found id: "9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff"
I0305 04:40:18.248299 2604473 cri.go:89] found id: "cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7"
I0305 04:40:18.248318 2604473 cri.go:89] found id: ""
I0305 04:40:18.248339 2604473 logs.go:276] 2 containers: [9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7]
I0305 04:40:18.248419 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:18.252542 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:18.257770 2604473 logs.go:123] Gathering logs for kube-scheduler [27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636] ...
I0305 04:40:18.257844 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636"
I0305 04:40:18.308816 2604473 logs.go:123] Gathering logs for container status ...
I0305 04:40:18.308886 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0305 04:40:18.377518 2604473 logs.go:123] Gathering logs for describe nodes ...
I0305 04:40:18.377651 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.20.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
I0305 04:40:18.562337 2604473 logs.go:123] Gathering logs for kube-apiserver [71cb2e24b42a7f8d797580bf0b66535d6aa909eae6127ef3f669c69251df7f53] ...
I0305 04:40:18.562481 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 71cb2e24b42a7f8d797580bf0b66535d6aa909eae6127ef3f669c69251df7f53"
I0305 04:40:18.645832 2604473 logs.go:123] Gathering logs for kindnet [767fb3c810f4e121fae9120786476ac6029d586fd1a6203f8cf8917dbbaf13cb] ...
I0305 04:40:18.645909 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 767fb3c810f4e121fae9120786476ac6029d586fd1a6203f8cf8917dbbaf13cb"
I0305 04:40:18.713062 2604473 logs.go:123] Gathering logs for storage-provisioner [cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7] ...
I0305 04:40:18.713194 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7"
I0305 04:40:18.767420 2604473 logs.go:123] Gathering logs for storage-provisioner [9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff] ...
I0305 04:40:18.767505 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff"
I0305 04:40:18.819044 2604473 logs.go:123] Gathering logs for kubelet ...
I0305 04:40:18.819120 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0305 04:40:18.885822 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566136 662 reflector.go:138] object-"kube-system"/"metrics-server-token-z55cz": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "metrics-server-token-z55cz" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.886097 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566265 662 reflector.go:138] object-"kube-system"/"kube-proxy-token-w7vc2": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kube-proxy-token-w7vc2" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.886324 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566376 662 reflector.go:138] object-"kube-system"/"coredns": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "coredns" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.886555 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566444 662 reflector.go:138] object-"kube-system"/"coredns-token-dkvsz": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "coredns-token-dkvsz" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.886787 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566500 662 reflector.go:138] object-"default"/"default-token-xt7xb": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "default-token-xt7xb" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "default": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.887035 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566558 662 reflector.go:138] object-"kube-system"/"storage-provisioner-token-hsz7w": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "storage-provisioner-token-hsz7w" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.887266 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566618 662 reflector.go:138] object-"kube-system"/"kindnet-token-jq4vl": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kindnet-token-jq4vl" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.887501 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566677 662 reflector.go:138] object-"kube-system"/"kube-proxy": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "kube-proxy" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.900193 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:51 old-k8s-version-929157 kubelet[662]: E0305 04:34:51.325447 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:18.900445 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:51 old-k8s-version-929157 kubelet[662]: E0305 04:34:51.878905 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.903428 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:05 old-k8s-version-929157 kubelet[662]: E0305 04:35:05.758082 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:18.905641 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:14 old-k8s-version-929157 kubelet[662]: E0305 04:35:14.030339 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.905991 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:15 old-k8s-version-929157 kubelet[662]: E0305 04:35:15.047564 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.906521 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:18 old-k8s-version-929157 kubelet[662]: E0305 04:35:18.750259 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.906865 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:22 old-k8s-version-929157 kubelet[662]: E0305 04:35:22.709995 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.907321 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:23 old-k8s-version-929157 kubelet[662]: E0305 04:35:23.063333 662 pod_workers.go:191] Error syncing pod b3db2690-165e-46e7-bef2-d3020d2c11e2 ("storage-provisioner_kube-system(b3db2690-165e-46e7-bef2-d3020d2c11e2)"), skipping: failed to "StartContainer" for "storage-provisioner" with CrashLoopBackOff: "back-off 10s restarting failed container=storage-provisioner pod=storage-provisioner_kube-system(b3db2690-165e-46e7-bef2-d3020d2c11e2)"
W0305 04:40:18.910113 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:30 old-k8s-version-929157 kubelet[662]: E0305 04:35:30.764359 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:18.910881 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:38 old-k8s-version-929157 kubelet[662]: E0305 04:35:38.133208 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.911226 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:42 old-k8s-version-929157 kubelet[662]: E0305 04:35:42.709684 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.911429 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:44 old-k8s-version-929157 kubelet[662]: E0305 04:35:44.750232 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.913473 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:53 old-k8s-version-929157 kubelet[662]: E0305 04:35:53.749653 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.913711 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:57 old-k8s-version-929157 kubelet[662]: E0305 04:35:57.760380 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.914318 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:08 old-k8s-version-929157 kubelet[662]: E0305 04:36:08.224516 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.914668 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:12 old-k8s-version-929157 kubelet[662]: E0305 04:36:12.710208 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.917119 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:13 old-k8s-version-929157 kubelet[662]: E0305 04:36:13.760058 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:18.917349 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:24 old-k8s-version-929157 kubelet[662]: E0305 04:36:24.750302 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.917692 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:27 old-k8s-version-929157 kubelet[662]: E0305 04:36:27.749613 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.917895 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:35 old-k8s-version-929157 kubelet[662]: E0305 04:36:35.750001 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.918239 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:38 old-k8s-version-929157 kubelet[662]: E0305 04:36:38.749574 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.918443 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:46 old-k8s-version-929157 kubelet[662]: E0305 04:36:46.750380 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.919044 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:51 old-k8s-version-929157 kubelet[662]: E0305 04:36:51.312622 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.919389 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:52 old-k8s-version-929157 kubelet[662]: E0305 04:36:52.709685 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.919609 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:01 old-k8s-version-929157 kubelet[662]: E0305 04:37:01.749989 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.919955 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:06 old-k8s-version-929157 kubelet[662]: E0305 04:37:06.750188 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.920159 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:12 old-k8s-version-929157 kubelet[662]: E0305 04:37:12.752686 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.920508 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:17 old-k8s-version-929157 kubelet[662]: E0305 04:37:17.749715 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.920714 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:25 old-k8s-version-929157 kubelet[662]: E0305 04:37:25.749931 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.921058 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:28 old-k8s-version-929157 kubelet[662]: E0305 04:37:28.749688 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.921406 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:39 old-k8s-version-929157 kubelet[662]: E0305 04:37:39.749886 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.923883 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:39 old-k8s-version-929157 kubelet[662]: E0305 04:37:39.757581 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:18.924233 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:50 old-k8s-version-929157 kubelet[662]: E0305 04:37:50.750942 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.924436 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:50 old-k8s-version-929157 kubelet[662]: E0305 04:37:50.751501 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.924782 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:02 old-k8s-version-929157 kubelet[662]: E0305 04:38:02.750200 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.924985 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:05 old-k8s-version-929157 kubelet[662]: E0305 04:38:05.750061 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.925588 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:18 old-k8s-version-929157 kubelet[662]: E0305 04:38:18.528409 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.925792 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:19 old-k8s-version-929157 kubelet[662]: E0305 04:38:19.750661 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.926140 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:22 old-k8s-version-929157 kubelet[662]: E0305 04:38:22.709687 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.926351 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:33 old-k8s-version-929157 kubelet[662]: E0305 04:38:33.749936 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.926697 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:36 old-k8s-version-929157 kubelet[662]: E0305 04:38:36.756212 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.926901 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:45 old-k8s-version-929157 kubelet[662]: E0305 04:38:45.749980 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.927248 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:48 old-k8s-version-929157 kubelet[662]: E0305 04:38:48.749677 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.928887 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:59 old-k8s-version-929157 kubelet[662]: E0305 04:38:59.750099 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.929263 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:03 old-k8s-version-929157 kubelet[662]: E0305 04:39:03.749594 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.929470 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:11 old-k8s-version-929157 kubelet[662]: E0305 04:39:11.749896 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.929818 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:15 old-k8s-version-929157 kubelet[662]: E0305 04:39:15.749647 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.930021 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:24 old-k8s-version-929157 kubelet[662]: E0305 04:39:24.750162 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.930364 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:26 old-k8s-version-929157 kubelet[662]: E0305 04:39:26.753574 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.930572 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:35 old-k8s-version-929157 kubelet[662]: E0305 04:39:35.750670 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.930917 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:39 old-k8s-version-929157 kubelet[662]: E0305 04:39:39.749986 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.931121 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:46 old-k8s-version-929157 kubelet[662]: E0305 04:39:46.750507 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.931566 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:51 old-k8s-version-929157 kubelet[662]: E0305 04:39:51.749596 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.931781 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:58 old-k8s-version-929157 kubelet[662]: E0305 04:39:58.753750 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.932124 2604473 logs.go:138] Found kubelet problem: Mar 05 04:40:03 old-k8s-version-929157 kubelet[662]: E0305 04:40:03.749667 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.932326 2604473 logs.go:138] Found kubelet problem: Mar 05 04:40:09 old-k8s-version-929157 kubelet[662]: E0305 04:40:09.750261 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.932680 2604473 logs.go:138] Found kubelet problem: Mar 05 04:40:17 old-k8s-version-929157 kubelet[662]: E0305 04:40:17.764535 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
I0305 04:40:18.932710 2604473 logs.go:123] Gathering logs for etcd [3dd18badcf58a90e38fe4db8a253226bf84cf241094215382ff4b9cc24ee04a5] ...
I0305 04:40:18.932738 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 3dd18badcf58a90e38fe4db8a253226bf84cf241094215382ff4b9cc24ee04a5"
I0305 04:40:19.010200 2604473 logs.go:123] Gathering logs for coredns [508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99] ...
I0305 04:40:19.010362 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99"
I0305 04:40:19.064284 2604473 logs.go:123] Gathering logs for kube-controller-manager [9a1fcc970baf72fae5fc3eb76833e0232c8f51d7938d48ea284de0940c3a961c] ...
I0305 04:40:19.064350 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 9a1fcc970baf72fae5fc3eb76833e0232c8f51d7938d48ea284de0940c3a961c"
I0305 04:40:19.127144 2604473 logs.go:123] Gathering logs for kube-controller-manager [e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341] ...
I0305 04:40:19.127178 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341"
I0305 04:40:19.217413 2604473 logs.go:123] Gathering logs for kubernetes-dashboard [4c1bed4d5ba29d0be56313d0a71a5ec6110f707ff84f3f6e34f4028a3d82e073] ...
I0305 04:40:19.217494 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 4c1bed4d5ba29d0be56313d0a71a5ec6110f707ff84f3f6e34f4028a3d82e073"
I0305 04:40:19.293327 2604473 logs.go:123] Gathering logs for kube-proxy [acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f] ...
I0305 04:40:19.293409 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f"
I0305 04:40:19.358594 2604473 logs.go:123] Gathering logs for kindnet [feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232] ...
I0305 04:40:19.358673 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232"
I0305 04:40:19.411252 2604473 logs.go:123] Gathering logs for dmesg ...
I0305 04:40:19.411335 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0305 04:40:19.430380 2604473 logs.go:123] Gathering logs for kube-apiserver [fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259] ...
I0305 04:40:19.430513 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259"
I0305 04:40:19.504401 2604473 logs.go:123] Gathering logs for etcd [35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605] ...
I0305 04:40:19.504432 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605"
I0305 04:40:19.555953 2604473 logs.go:123] Gathering logs for coredns [e0dbacaa8e30885944ab46b688f9bbefac308f823788712e6c14ecdc1556f552] ...
I0305 04:40:19.556023 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 e0dbacaa8e30885944ab46b688f9bbefac308f823788712e6c14ecdc1556f552"
I0305 04:40:19.606777 2604473 logs.go:123] Gathering logs for kube-scheduler [a5d43d0d8a5d3f1e42ca4c1b9ec4dd6a812c6454f3858ec238850f1190bce93c] ...
I0305 04:40:19.606856 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 a5d43d0d8a5d3f1e42ca4c1b9ec4dd6a812c6454f3858ec238850f1190bce93c"
I0305 04:40:19.655430 2604473 logs.go:123] Gathering logs for kube-proxy [dd4135e2a8d47ee5af337a3314218c302ed97e8f2c878263df59d7b367ea4443] ...
I0305 04:40:19.655537 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 dd4135e2a8d47ee5af337a3314218c302ed97e8f2c878263df59d7b367ea4443"
I0305 04:40:19.712578 2604473 logs.go:123] Gathering logs for containerd ...
I0305 04:40:19.712646 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0305 04:40:19.792651 2604473 out.go:304] Setting ErrFile to fd 2...
I0305 04:40:19.792908 2604473 out.go:338] TERM=,COLORTERM=, which probably does not support color
W0305 04:40:19.792978 2604473 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0305 04:40:19.793013 2604473 out.go:239] Mar 05 04:39:51 old-k8s-version-929157 kubelet[662]: E0305 04:39:51.749596 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
Mar 05 04:39:51 old-k8s-version-929157 kubelet[662]: E0305 04:39:51.749596 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:19.793047 2604473 out.go:239] Mar 05 04:39:58 old-k8s-version-929157 kubelet[662]: E0305 04:39:58.753750 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Mar 05 04:39:58 old-k8s-version-929157 kubelet[662]: E0305 04:39:58.753750 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:19.793090 2604473 out.go:239] Mar 05 04:40:03 old-k8s-version-929157 kubelet[662]: E0305 04:40:03.749667 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
Mar 05 04:40:03 old-k8s-version-929157 kubelet[662]: E0305 04:40:03.749667 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:19.793123 2604473 out.go:239] Mar 05 04:40:09 old-k8s-version-929157 kubelet[662]: E0305 04:40:09.750261 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Mar 05 04:40:09 old-k8s-version-929157 kubelet[662]: E0305 04:40:09.750261 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:19.793152 2604473 out.go:239] Mar 05 04:40:17 old-k8s-version-929157 kubelet[662]: E0305 04:40:17.764535 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
Mar 05 04:40:17 old-k8s-version-929157 kubelet[662]: E0305 04:40:17.764535 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
I0305 04:40:19.793189 2604473 out.go:304] Setting ErrFile to fd 2...
I0305 04:40:19.793210 2604473 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0305 04:40:29.793931 2604473 api_server.go:253] Checking apiserver healthz at https://192.168.76.2:8443/healthz ...
I0305 04:40:29.810892 2604473 api_server.go:279] https://192.168.76.2:8443/healthz returned 200:
ok
I0305 04:40:29.813726 2604473 out.go:177]
W0305 04:40:29.815636 2604473 out.go:239] X Exiting due to K8S_UNHEALTHY_CONTROL_PLANE: wait 6m0s for node: wait for healthy API server: controlPlane never updated to v1.20.0
X Exiting due to K8S_UNHEALTHY_CONTROL_PLANE: wait 6m0s for node: wait for healthy API server: controlPlane never updated to v1.20.0
W0305 04:40:29.815818 2604473 out.go:239] * Suggestion: Control Plane could not update, try minikube delete --all --purge
* Suggestion: Control Plane could not update, try minikube delete --all --purge
W0305 04:40:29.815873 2604473 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/11417
* Related issue: https://github.com/kubernetes/minikube/issues/11417
W0305 04:40:29.815906 2604473 out.go:239] *
*
W0305 04:40:29.817379 2604473 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0305 04:40:29.820417 2604473 out.go:177]
** /stderr **
start_stop_delete_test.go:259: failed to start minikube post-stop. args "out/minikube-linux-arm64 start -p old-k8s-version-929157 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --container-runtime=containerd --kubernetes-version=v1.20.0": exit status 102
helpers_test.go:222: -----------------------post-mortem--------------------------------
helpers_test.go:230: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/SecondStart]: docker inspect <======
helpers_test.go:231: (dbg) Run: docker inspect old-k8s-version-929157
helpers_test.go:235: (dbg) docker inspect old-k8s-version-929157:
-- stdout --
[
{
"Id": "c652382b08aaa0c43f124d126b0a211400adccfcfb6dfb673f1a7669cc172761",
"Created": "2024-03-05T04:31:10.614136255Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2604668,
"ExitCode": 0,
"Error": "",
"StartedAt": "2024-03-05T04:34:21.431201835Z",
"FinishedAt": "2024-03-05T04:34:19.783259881Z"
},
"Image": "sha256:4a9b65157dd7fb2ddb7cb7afe975b3dc288e9877c60d13613a69dd41a70e2e4e",
"ResolvConfPath": "/var/lib/docker/containers/c652382b08aaa0c43f124d126b0a211400adccfcfb6dfb673f1a7669cc172761/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/c652382b08aaa0c43f124d126b0a211400adccfcfb6dfb673f1a7669cc172761/hostname",
"HostsPath": "/var/lib/docker/containers/c652382b08aaa0c43f124d126b0a211400adccfcfb6dfb673f1a7669cc172761/hosts",
"LogPath": "/var/lib/docker/containers/c652382b08aaa0c43f124d126b0a211400adccfcfb6dfb673f1a7669cc172761/c652382b08aaa0c43f124d126b0a211400adccfcfb6dfb673f1a7669cc172761-json.log",
"Name": "/old-k8s-version-929157",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "unconfined",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro",
"old-k8s-version-929157:/var"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "old-k8s-version-929157",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "host",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 2306867200,
"NanoCpus": 2000000000,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": [],
"BlkioDeviceWriteBps": [],
"BlkioDeviceReadIOps": [],
"BlkioDeviceWriteIOps": [],
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 4613734400,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": [],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/42dd374c19339c3a421f50c064eb5192f5389e9bc6cbf15a1e60e0488cf118a6-init/diff:/var/lib/docker/overlay2/34e036c1fb135ad053cbc538122e3cc2a8a5ea3b218e432eccf1b69178654f4d/diff",
"MergedDir": "/var/lib/docker/overlay2/42dd374c19339c3a421f50c064eb5192f5389e9bc6cbf15a1e60e0488cf118a6/merged",
"UpperDir": "/var/lib/docker/overlay2/42dd374c19339c3a421f50c064eb5192f5389e9bc6cbf15a1e60e0488cf118a6/diff",
"WorkDir": "/var/lib/docker/overlay2/42dd374c19339c3a421f50c064eb5192f5389e9bc6cbf15a1e60e0488cf118a6/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "old-k8s-version-929157",
"Source": "/var/lib/docker/volumes/old-k8s-version-929157/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "old-k8s-version-929157",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"22/tcp": {},
"2376/tcp": {},
"32443/tcp": {},
"5000/tcp": {},
"8443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "old-k8s-version-929157",
"name.minikube.sigs.k8s.io": "old-k8s-version-929157",
"role.minikube.sigs.k8s.io": ""
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "b3de2ba3f535a23900625341ff68db81549470090ed30e4841b7e197966822c6",
"SandboxKey": "/var/run/docker/netns/b3de2ba3f535",
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "35352"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "35351"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "35348"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "35350"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "35349"
}
]
},
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"old-k8s-version-929157": {
"IPAMConfig": {
"IPv4Address": "192.168.76.2"
},
"Links": null,
"Aliases": [
"c652382b08aa",
"old-k8s-version-929157"
],
"MacAddress": "02:42:c0:a8:4c:02",
"NetworkID": "6ec5655ad76a618340e4f29b093f2888fef6e855529301f9276b9208f58e3cde",
"EndpointID": "e102476cac4346f171fdf13c41cb51b5065c5a076014b9fff76062b50dc10ca2",
"Gateway": "192.168.76.1",
"IPAddress": "192.168.76.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"DriverOpts": null,
"DNSNames": [
"old-k8s-version-929157",
"c652382b08aa"
]
}
}
}
}
]
-- /stdout --
helpers_test.go:239: (dbg) Run: out/minikube-linux-arm64 status --format={{.Host}} -p old-k8s-version-929157 -n old-k8s-version-929157
helpers_test.go:244: <<< TestStartStop/group/old-k8s-version/serial/SecondStart FAILED: start of post-mortem logs <<<
helpers_test.go:245: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/SecondStart]: minikube logs <======
helpers_test.go:247: (dbg) Run: out/minikube-linux-arm64 -p old-k8s-version-929157 logs -n 25
helpers_test.go:247: (dbg) Done: out/minikube-linux-arm64 -p old-k8s-version-929157 logs -n 25: (3.324715553s)
helpers_test.go:252: TestStartStop/group/old-k8s-version/serial/SecondStart logs:
-- stdout --
==> Audit <==
|---------|--------------------------------------------------------|--------------------------|---------|---------|---------------------|---------------------|
| Command | Args | Profile | User | Version | Start Time | End Time |
|---------|--------------------------------------------------------|--------------------------|---------|---------|---------------------|---------------------|
| start | -p cert-expiration-103762 | cert-expiration-103762 | jenkins | v1.32.0 | 05 Mar 24 04:29 UTC | 05 Mar 24 04:30 UTC |
| | --memory=2048 | | | | | |
| | --cert-expiration=3m | | | | | |
| | --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| ssh | force-systemd-env-042508 | force-systemd-env-042508 | jenkins | v1.32.0 | 05 Mar 24 04:30 UTC | 05 Mar 24 04:30 UTC |
| | ssh cat | | | | | |
| | /etc/containerd/config.toml | | | | | |
| delete | -p force-systemd-env-042508 | force-systemd-env-042508 | jenkins | v1.32.0 | 05 Mar 24 04:30 UTC | 05 Mar 24 04:30 UTC |
| start | -p cert-options-410621 | cert-options-410621 | jenkins | v1.32.0 | 05 Mar 24 04:30 UTC | 05 Mar 24 04:31 UTC |
| | --memory=2048 | | | | | |
| | --apiserver-ips=127.0.0.1 | | | | | |
| | --apiserver-ips=192.168.15.15 | | | | | |
| | --apiserver-names=localhost | | | | | |
| | --apiserver-names=www.google.com | | | | | |
| | --apiserver-port=8555 | | | | | |
| | --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| ssh | cert-options-410621 ssh | cert-options-410621 | jenkins | v1.32.0 | 05 Mar 24 04:31 UTC | 05 Mar 24 04:31 UTC |
| | openssl x509 -text -noout -in | | | | | |
| | /var/lib/minikube/certs/apiserver.crt | | | | | |
| ssh | -p cert-options-410621 -- sudo | cert-options-410621 | jenkins | v1.32.0 | 05 Mar 24 04:31 UTC | 05 Mar 24 04:31 UTC |
| | cat /etc/kubernetes/admin.conf | | | | | |
| delete | -p cert-options-410621 | cert-options-410621 | jenkins | v1.32.0 | 05 Mar 24 04:31 UTC | 05 Mar 24 04:31 UTC |
| start | -p old-k8s-version-929157 | old-k8s-version-929157 | jenkins | v1.32.0 | 05 Mar 24 04:31 UTC | 05 Mar 24 04:33 UTC |
| | --memory=2200 | | | | | |
| | --alsologtostderr --wait=true | | | | | |
| | --kvm-network=default | | | | | |
| | --kvm-qemu-uri=qemu:///system | | | | | |
| | --disable-driver-mounts | | | | | |
| | --keep-context=false | | | | | |
| | --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| | --kubernetes-version=v1.20.0 | | | | | |
| start | -p cert-expiration-103762 | cert-expiration-103762 | jenkins | v1.32.0 | 05 Mar 24 04:33 UTC | 05 Mar 24 04:33 UTC |
| | --memory=2048 | | | | | |
| | --cert-expiration=8760h | | | | | |
| | --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| delete | -p cert-expiration-103762 | cert-expiration-103762 | jenkins | v1.32.0 | 05 Mar 24 04:33 UTC | 05 Mar 24 04:33 UTC |
| start | -p no-preload-296091 | no-preload-296091 | jenkins | v1.32.0 | 05 Mar 24 04:33 UTC | 05 Mar 24 04:34 UTC |
| | --memory=2200 --alsologtostderr | | | | | |
| | --wait=true --preload=false | | | | | |
| | --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| | --kubernetes-version=v1.29.0-rc.2 | | | | | |
| addons | enable metrics-server -p old-k8s-version-929157 | old-k8s-version-929157 | jenkins | v1.32.0 | 05 Mar 24 04:34 UTC | 05 Mar 24 04:34 UTC |
| | --images=MetricsServer=registry.k8s.io/echoserver:1.4 | | | | | |
| | --registries=MetricsServer=fake.domain | | | | | |
| stop | -p old-k8s-version-929157 | old-k8s-version-929157 | jenkins | v1.32.0 | 05 Mar 24 04:34 UTC | 05 Mar 24 04:34 UTC |
| | --alsologtostderr -v=3 | | | | | |
| addons | enable dashboard -p old-k8s-version-929157 | old-k8s-version-929157 | jenkins | v1.32.0 | 05 Mar 24 04:34 UTC | 05 Mar 24 04:34 UTC |
| | --images=MetricsScraper=registry.k8s.io/echoserver:1.4 | | | | | |
| start | -p old-k8s-version-929157 | old-k8s-version-929157 | jenkins | v1.32.0 | 05 Mar 24 04:34 UTC | |
| | --memory=2200 | | | | | |
| | --alsologtostderr --wait=true | | | | | |
| | --kvm-network=default | | | | | |
| | --kvm-qemu-uri=qemu:///system | | | | | |
| | --disable-driver-mounts | | | | | |
| | --keep-context=false | | | | | |
| | --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| | --kubernetes-version=v1.20.0 | | | | | |
| addons | enable metrics-server -p no-preload-296091 | no-preload-296091 | jenkins | v1.32.0 | 05 Mar 24 04:35 UTC | 05 Mar 24 04:35 UTC |
| | --images=MetricsServer=registry.k8s.io/echoserver:1.4 | | | | | |
| | --registries=MetricsServer=fake.domain | | | | | |
| stop | -p no-preload-296091 | no-preload-296091 | jenkins | v1.32.0 | 05 Mar 24 04:35 UTC | 05 Mar 24 04:35 UTC |
| | --alsologtostderr -v=3 | | | | | |
| addons | enable dashboard -p no-preload-296091 | no-preload-296091 | jenkins | v1.32.0 | 05 Mar 24 04:35 UTC | 05 Mar 24 04:35 UTC |
| | --images=MetricsScraper=registry.k8s.io/echoserver:1.4 | | | | | |
| start | -p no-preload-296091 | no-preload-296091 | jenkins | v1.32.0 | 05 Mar 24 04:35 UTC | 05 Mar 24 04:39 UTC |
| | --memory=2200 --alsologtostderr | | | | | |
| | --wait=true --preload=false | | | | | |
| | --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| | --kubernetes-version=v1.29.0-rc.2 | | | | | |
| image | no-preload-296091 image list | no-preload-296091 | jenkins | v1.32.0 | 05 Mar 24 04:39 UTC | 05 Mar 24 04:39 UTC |
| | --format=json | | | | | |
| pause | -p no-preload-296091 | no-preload-296091 | jenkins | v1.32.0 | 05 Mar 24 04:39 UTC | 05 Mar 24 04:39 UTC |
| | --alsologtostderr -v=1 | | | | | |
| unpause | -p no-preload-296091 | no-preload-296091 | jenkins | v1.32.0 | 05 Mar 24 04:40 UTC | 05 Mar 24 04:40 UTC |
| | --alsologtostderr -v=1 | | | | | |
| delete | -p no-preload-296091 | no-preload-296091 | jenkins | v1.32.0 | 05 Mar 24 04:40 UTC | 05 Mar 24 04:40 UTC |
| delete | -p no-preload-296091 | no-preload-296091 | jenkins | v1.32.0 | 05 Mar 24 04:40 UTC | 05 Mar 24 04:40 UTC |
| start | -p embed-certs-046985 | embed-certs-046985 | jenkins | v1.32.0 | 05 Mar 24 04:40 UTC | |
| | --memory=2200 | | | | | |
| | --alsologtostderr --wait=true | | | | | |
| | --embed-certs --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| | --kubernetes-version=v1.28.4 | | | | | |
|---------|--------------------------------------------------------|--------------------------|---------|---------|---------------------|---------------------|
==> Last Start <==
Log file created at: 2024/03/05 04:40:05
Running on machine: ip-172-31-30-239
Binary: Built with gc go1.22.0 for linux/arm64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0305 04:40:05.896195 2614612 out.go:291] Setting OutFile to fd 1 ...
I0305 04:40:05.896403 2614612 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0305 04:40:05.896430 2614612 out.go:304] Setting ErrFile to fd 2...
I0305 04:40:05.896449 2614612 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0305 04:40:05.896732 2614612 root.go:338] Updating PATH: /home/jenkins/minikube-integration/17909-2401808/.minikube/bin
I0305 04:40:05.897186 2614612 out.go:298] Setting JSON to false
I0305 04:40:05.900460 2614612 start.go:129] hostinfo: {"hostname":"ip-172-31-30-239","uptime":37351,"bootTime":1709576255,"procs":220,"os":"linux","platform":"ubuntu","platformFamily":"debian","platformVersion":"20.04","kernelVersion":"5.15.0-1055-aws","kernelArch":"aarch64","virtualizationSystem":"","virtualizationRole":"","hostId":"92f46a7d-c249-4c12-924a-77f64874c910"}
I0305 04:40:05.900589 2614612 start.go:139] virtualization:
I0305 04:40:05.903623 2614612 out.go:177] * [embed-certs-046985] minikube v1.32.0 on Ubuntu 20.04 (arm64)
I0305 04:40:05.905939 2614612 out.go:177] - MINIKUBE_LOCATION=17909
I0305 04:40:05.908185 2614612 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0305 04:40:05.906125 2614612 notify.go:220] Checking for updates...
I0305 04:40:05.912879 2614612 out.go:177] - KUBECONFIG=/home/jenkins/minikube-integration/17909-2401808/kubeconfig
I0305 04:40:05.914829 2614612 out.go:177] - MINIKUBE_HOME=/home/jenkins/minikube-integration/17909-2401808/.minikube
I0305 04:40:05.916762 2614612 out.go:177] - MINIKUBE_BIN=out/minikube-linux-arm64
I0305 04:40:05.918930 2614612 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0305 04:40:05.921488 2614612 config.go:182] Loaded profile config "old-k8s-version-929157": Driver=docker, ContainerRuntime=containerd, KubernetesVersion=v1.20.0
I0305 04:40:05.921595 2614612 driver.go:392] Setting default libvirt URI to qemu:///system
I0305 04:40:05.957445 2614612 docker.go:122] docker version: linux-25.0.3:Docker Engine - Community
I0305 04:40:05.957563 2614612 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0305 04:40:06.069545 2614612 info.go:266] docker info: {ID:6ZPO:QZND:VNGE:LUKL:4Y3K:XELL:AAX4:2GTK:E6LM:MPRN:3ZXR:TTMR Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:32 OomKillDisable:true NGoroutines:56 SystemTime:2024-03-05 04:40:06.058617484 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1055-aws OperatingSystem:Ubuntu 20.04.6 LTS OSType:linux Architecture:aar
ch64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:2 MemTotal:8215040000 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ip-172-31-30-239 Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=builtin] ProductLicense: Warnings:<nil> ServerErro
rs:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.6]] Warnings:<nil>}}
I0305 04:40:06.069658 2614612 docker.go:295] overlay module found
I0305 04:40:06.072143 2614612 out.go:177] * Using the docker driver based on user configuration
I0305 04:40:06.074345 2614612 start.go:297] selected driver: docker
I0305 04:40:06.074368 2614612 start.go:901] validating driver "docker" against <nil>
I0305 04:40:06.074383 2614612 start.go:912] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0305 04:40:06.075054 2614612 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0305 04:40:06.179039 2614612 info.go:266] docker info: {ID:6ZPO:QZND:VNGE:LUKL:4Y3K:XELL:AAX4:2GTK:E6LM:MPRN:3ZXR:TTMR Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:32 OomKillDisable:true NGoroutines:56 SystemTime:2024-03-05 04:40:06.169813897 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1055-aws OperatingSystem:Ubuntu 20.04.6 LTS OSType:linux Architecture:aar
ch64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:2 MemTotal:8215040000 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ip-172-31-30-239 Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=builtin] ProductLicense: Warnings:<nil> ServerErro
rs:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.6]] Warnings:<nil>}}
I0305 04:40:06.179257 2614612 start_flags.go:310] no existing cluster config was found, will generate one from the flags
I0305 04:40:06.181169 2614612 start_flags.go:947] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0305 04:40:06.191773 2614612 out.go:177] * Using Docker driver with root privileges
I0305 04:40:06.193609 2614612 cni.go:84] Creating CNI manager for ""
I0305 04:40:06.193651 2614612 cni.go:143] "docker" driver + "containerd" runtime found, recommending kindnet
I0305 04:40:06.193668 2614612 start_flags.go:319] Found "CNI" CNI - setting NetworkPlugin=cni
I0305 04:40:06.193765 2614612 start.go:340] cluster config:
{Name:embed-certs-046985 KeepContext:false EmbedCerts:true MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.28.4 ClusterName:embed-certs-046985 Namespace:default APIServerHAVIP: APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local Contain
erRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.28.4 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: Stati
cIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0305 04:40:06.196437 2614612 out.go:177] * Starting "embed-certs-046985" primary control-plane node in "embed-certs-046985" cluster
I0305 04:40:06.198481 2614612 cache.go:121] Beginning downloading kic base image for docker with containerd
I0305 04:40:06.200587 2614612 out.go:177] * Pulling base image v0.0.42-1708944392-18244 ...
I0305 04:40:06.202713 2614612 preload.go:132] Checking if preload exists for k8s version v1.28.4 and runtime containerd
I0305 04:40:06.202768 2614612 preload.go:147] Found local preload: /home/jenkins/minikube-integration/17909-2401808/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.28.4-containerd-overlay2-arm64.tar.lz4
I0305 04:40:06.202777 2614612 cache.go:56] Caching tarball of preloaded images
I0305 04:40:06.202857 2614612 preload.go:173] Found /home/jenkins/minikube-integration/17909-2401808/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.28.4-containerd-overlay2-arm64.tar.lz4 in cache, skipping download
I0305 04:40:06.202867 2614612 cache.go:59] Finished verifying existence of preloaded tar for v1.28.4 on containerd
I0305 04:40:06.202982 2614612 profile.go:142] Saving config to /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/config.json ...
I0305 04:40:06.202999 2614612 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/config.json: {Name:mk73d2212526b29dee5a1c9d392417b39ebd3e1d Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0305 04:40:06.202690 2614612 image.go:79] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 in local docker daemon
I0305 04:40:06.238517 2614612 image.go:83] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 in local docker daemon, skipping pull
I0305 04:40:06.238557 2614612 cache.go:144] gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 exists in daemon, skipping load
I0305 04:40:06.238586 2614612 cache.go:194] Successfully downloaded all kic artifacts
I0305 04:40:06.238632 2614612 start.go:360] acquireMachinesLock for embed-certs-046985: {Name:mkec07e7d3d1859ddd9db7f37a66b60a7e62958c Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0305 04:40:06.238892 2614612 start.go:364] duration metric: took 230.197µs to acquireMachinesLock for "embed-certs-046985"
I0305 04:40:06.238936 2614612 start.go:93] Provisioning new machine with config: &{Name:embed-certs-046985 KeepContext:false EmbedCerts:true MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.28.4 ClusterName:embed-certs-046985 Namespace:default APIServerHAVIP: APIServe
rName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.28.4 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableM
etrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s} &{Name: IP: Port:8443 KubernetesVersion:v1.28.4 ContainerRuntime:containerd ControlPlane:true Worker:true}
I0305 04:40:06.239043 2614612 start.go:125] createHost starting for "" (driver="docker")
I0305 04:40:05.626979 2604473 cri.go:89] found id: "9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff"
I0305 04:40:05.627006 2604473 cri.go:89] found id: "cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7"
I0305 04:40:05.627012 2604473 cri.go:89] found id: ""
I0305 04:40:05.627020 2604473 logs.go:276] 2 containers: [9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7]
I0305 04:40:05.627097 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.631684 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:05.635856 2604473 logs.go:123] Gathering logs for containerd ...
I0305 04:40:05.635888 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0305 04:40:05.707408 2604473 logs.go:123] Gathering logs for kubelet ...
I0305 04:40:05.707485 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0305 04:40:05.768492 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566136 662 reflector.go:138] object-"kube-system"/"metrics-server-token-z55cz": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "metrics-server-token-z55cz" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.768846 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566265 662 reflector.go:138] object-"kube-system"/"kube-proxy-token-w7vc2": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kube-proxy-token-w7vc2" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.769080 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566376 662 reflector.go:138] object-"kube-system"/"coredns": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "coredns" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.769316 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566444 662 reflector.go:138] object-"kube-system"/"coredns-token-dkvsz": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "coredns-token-dkvsz" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.769553 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566500 662 reflector.go:138] object-"default"/"default-token-xt7xb": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "default-token-xt7xb" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "default": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.769802 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566558 662 reflector.go:138] object-"kube-system"/"storage-provisioner-token-hsz7w": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "storage-provisioner-token-hsz7w" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.770040 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566618 662 reflector.go:138] object-"kube-system"/"kindnet-token-jq4vl": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kindnet-token-jq4vl" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.770266 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566677 662 reflector.go:138] object-"kube-system"/"kube-proxy": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "kube-proxy" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:05.779863 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:51 old-k8s-version-929157 kubelet[662]: E0305 04:34:51.325447 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:05.780103 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:51 old-k8s-version-929157 kubelet[662]: E0305 04:34:51.878905 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.783162 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:05 old-k8s-version-929157 kubelet[662]: E0305 04:35:05.758082 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:05.785363 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:14 old-k8s-version-929157 kubelet[662]: E0305 04:35:14.030339 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.785830 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:15 old-k8s-version-929157 kubelet[662]: E0305 04:35:15.047564 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.786350 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:18 old-k8s-version-929157 kubelet[662]: E0305 04:35:18.750259 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.786679 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:22 old-k8s-version-929157 kubelet[662]: E0305 04:35:22.709995 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.787115 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:23 old-k8s-version-929157 kubelet[662]: E0305 04:35:23.063333 662 pod_workers.go:191] Error syncing pod b3db2690-165e-46e7-bef2-d3020d2c11e2 ("storage-provisioner_kube-system(b3db2690-165e-46e7-bef2-d3020d2c11e2)"), skipping: failed to "StartContainer" for "storage-provisioner" with CrashLoopBackOff: "back-off 10s restarting failed container=storage-provisioner pod=storage-provisioner_kube-system(b3db2690-165e-46e7-bef2-d3020d2c11e2)"
W0305 04:40:05.790026 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:30 old-k8s-version-929157 kubelet[662]: E0305 04:35:30.764359 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:05.790813 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:38 old-k8s-version-929157 kubelet[662]: E0305 04:35:38.133208 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.791171 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:42 old-k8s-version-929157 kubelet[662]: E0305 04:35:42.709684 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.791380 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:44 old-k8s-version-929157 kubelet[662]: E0305 04:35:44.750232 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.791756 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:53 old-k8s-version-929157 kubelet[662]: E0305 04:35:53.749653 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.791968 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:57 old-k8s-version-929157 kubelet[662]: E0305 04:35:57.760380 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.792575 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:08 old-k8s-version-929157 kubelet[662]: E0305 04:36:08.224516 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.792923 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:12 old-k8s-version-929157 kubelet[662]: E0305 04:36:12.710208 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.795355 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:13 old-k8s-version-929157 kubelet[662]: E0305 04:36:13.760058 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:05.795615 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:24 old-k8s-version-929157 kubelet[662]: E0305 04:36:24.750302 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.796035 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:27 old-k8s-version-929157 kubelet[662]: E0305 04:36:27.749613 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.796286 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:35 old-k8s-version-929157 kubelet[662]: E0305 04:36:35.750001 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.796707 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:38 old-k8s-version-929157 kubelet[662]: E0305 04:36:38.749574 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.797944 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:46 old-k8s-version-929157 kubelet[662]: E0305 04:36:46.750380 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.798615 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:51 old-k8s-version-929157 kubelet[662]: E0305 04:36:51.312622 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.798943 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:52 old-k8s-version-929157 kubelet[662]: E0305 04:36:52.709685 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.799128 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:01 old-k8s-version-929157 kubelet[662]: E0305 04:37:01.749989 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.799463 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:06 old-k8s-version-929157 kubelet[662]: E0305 04:37:06.750188 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.799655 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:12 old-k8s-version-929157 kubelet[662]: E0305 04:37:12.752686 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.799981 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:17 old-k8s-version-929157 kubelet[662]: E0305 04:37:17.749715 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.800160 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:25 old-k8s-version-929157 kubelet[662]: E0305 04:37:25.749931 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.800482 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:28 old-k8s-version-929157 kubelet[662]: E0305 04:37:28.749688 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.800809 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:39 old-k8s-version-929157 kubelet[662]: E0305 04:37:39.749886 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.803220 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:39 old-k8s-version-929157 kubelet[662]: E0305 04:37:39.757581 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:05.803602 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:50 old-k8s-version-929157 kubelet[662]: E0305 04:37:50.750942 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.803822 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:50 old-k8s-version-929157 kubelet[662]: E0305 04:37:50.751501 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.804195 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:02 old-k8s-version-929157 kubelet[662]: E0305 04:38:02.750200 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.804439 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:05 old-k8s-version-929157 kubelet[662]: E0305 04:38:05.750061 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.805100 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:18 old-k8s-version-929157 kubelet[662]: E0305 04:38:18.528409 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.805318 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:19 old-k8s-version-929157 kubelet[662]: E0305 04:38:19.750661 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.805692 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:22 old-k8s-version-929157 kubelet[662]: E0305 04:38:22.709687 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.805909 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:33 old-k8s-version-929157 kubelet[662]: E0305 04:38:33.749936 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.806279 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:36 old-k8s-version-929157 kubelet[662]: E0305 04:38:36.756212 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.806486 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:45 old-k8s-version-929157 kubelet[662]: E0305 04:38:45.749980 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.806868 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:48 old-k8s-version-929157 kubelet[662]: E0305 04:38:48.749677 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.807073 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:59 old-k8s-version-929157 kubelet[662]: E0305 04:38:59.750099 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.807464 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:03 old-k8s-version-929157 kubelet[662]: E0305 04:39:03.749594 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.807748 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:11 old-k8s-version-929157 kubelet[662]: E0305 04:39:11.749896 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.808093 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:15 old-k8s-version-929157 kubelet[662]: E0305 04:39:15.749647 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.808322 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:24 old-k8s-version-929157 kubelet[662]: E0305 04:39:24.750162 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.808670 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:26 old-k8s-version-929157 kubelet[662]: E0305 04:39:26.753574 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.808879 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:35 old-k8s-version-929157 kubelet[662]: E0305 04:39:35.750670 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.809232 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:39 old-k8s-version-929157 kubelet[662]: E0305 04:39:39.749986 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.809437 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:46 old-k8s-version-929157 kubelet[662]: E0305 04:39:46.750507 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.809789 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:51 old-k8s-version-929157 kubelet[662]: E0305 04:39:51.749596 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:05.809997 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:58 old-k8s-version-929157 kubelet[662]: E0305 04:39:58.753750 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:05.810345 2604473 logs.go:138] Found kubelet problem: Mar 05 04:40:03 old-k8s-version-929157 kubelet[662]: E0305 04:40:03.749667 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
I0305 04:40:05.810369 2604473 logs.go:123] Gathering logs for kube-apiserver [fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259] ...
I0305 04:40:05.810399 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259"
I0305 04:40:05.903991 2604473 logs.go:123] Gathering logs for coredns [e0dbacaa8e30885944ab46b688f9bbefac308f823788712e6c14ecdc1556f552] ...
I0305 04:40:05.904037 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 e0dbacaa8e30885944ab46b688f9bbefac308f823788712e6c14ecdc1556f552"
I0305 04:40:05.973325 2604473 logs.go:123] Gathering logs for kube-scheduler [a5d43d0d8a5d3f1e42ca4c1b9ec4dd6a812c6454f3858ec238850f1190bce93c] ...
I0305 04:40:05.973352 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 a5d43d0d8a5d3f1e42ca4c1b9ec4dd6a812c6454f3858ec238850f1190bce93c"
I0305 04:40:06.039539 2604473 logs.go:123] Gathering logs for kube-scheduler [27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636] ...
I0305 04:40:06.039576 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636"
I0305 04:40:06.101871 2604473 logs.go:123] Gathering logs for kube-controller-manager [e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341] ...
I0305 04:40:06.101965 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341"
I0305 04:40:06.208749 2604473 logs.go:123] Gathering logs for kindnet [767fb3c810f4e121fae9120786476ac6029d586fd1a6203f8cf8917dbbaf13cb] ...
I0305 04:40:06.208798 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 767fb3c810f4e121fae9120786476ac6029d586fd1a6203f8cf8917dbbaf13cb"
I0305 04:40:06.265134 2604473 logs.go:123] Gathering logs for coredns [508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99] ...
I0305 04:40:06.265559 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99"
I0305 04:40:06.326851 2604473 logs.go:123] Gathering logs for kube-proxy [dd4135e2a8d47ee5af337a3314218c302ed97e8f2c878263df59d7b367ea4443] ...
I0305 04:40:06.326882 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 dd4135e2a8d47ee5af337a3314218c302ed97e8f2c878263df59d7b367ea4443"
I0305 04:40:06.402071 2604473 logs.go:123] Gathering logs for kube-proxy [acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f] ...
I0305 04:40:06.402097 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f"
I0305 04:40:06.477887 2604473 logs.go:123] Gathering logs for kube-controller-manager [9a1fcc970baf72fae5fc3eb76833e0232c8f51d7938d48ea284de0940c3a961c] ...
I0305 04:40:06.477930 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 9a1fcc970baf72fae5fc3eb76833e0232c8f51d7938d48ea284de0940c3a961c"
I0305 04:40:06.556386 2604473 logs.go:123] Gathering logs for container status ...
I0305 04:40:06.556423 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0305 04:40:06.640735 2604473 logs.go:123] Gathering logs for dmesg ...
I0305 04:40:06.640819 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0305 04:40:06.660696 2604473 logs.go:123] Gathering logs for describe nodes ...
I0305 04:40:06.660771 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.20.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
I0305 04:40:06.935963 2604473 logs.go:123] Gathering logs for etcd [3dd18badcf58a90e38fe4db8a253226bf84cf241094215382ff4b9cc24ee04a5] ...
I0305 04:40:06.936044 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 3dd18badcf58a90e38fe4db8a253226bf84cf241094215382ff4b9cc24ee04a5"
I0305 04:40:07.014969 2604473 logs.go:123] Gathering logs for etcd [35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605] ...
I0305 04:40:07.017390 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605"
I0305 04:40:07.120671 2604473 logs.go:123] Gathering logs for kindnet [feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232] ...
I0305 04:40:07.120750 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232"
I0305 04:40:07.165436 2604473 logs.go:123] Gathering logs for storage-provisioner [9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff] ...
I0305 04:40:07.165514 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff"
I0305 04:40:07.244226 2604473 logs.go:123] Gathering logs for storage-provisioner [cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7] ...
I0305 04:40:07.244266 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7"
I0305 04:40:07.306233 2604473 logs.go:123] Gathering logs for kube-apiserver [71cb2e24b42a7f8d797580bf0b66535d6aa909eae6127ef3f669c69251df7f53] ...
I0305 04:40:07.306259 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 71cb2e24b42a7f8d797580bf0b66535d6aa909eae6127ef3f669c69251df7f53"
I0305 04:40:07.442298 2604473 logs.go:123] Gathering logs for kubernetes-dashboard [4c1bed4d5ba29d0be56313d0a71a5ec6110f707ff84f3f6e34f4028a3d82e073] ...
I0305 04:40:07.442336 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 4c1bed4d5ba29d0be56313d0a71a5ec6110f707ff84f3f6e34f4028a3d82e073"
I0305 04:40:07.518486 2604473 out.go:304] Setting ErrFile to fd 2...
I0305 04:40:07.518513 2604473 out.go:338] TERM=,COLORTERM=, which probably does not support color
W0305 04:40:07.518562 2604473 out.go:239] X Problems detected in kubelet:
W0305 04:40:07.518575 2604473 out.go:239] Mar 05 04:39:39 old-k8s-version-929157 kubelet[662]: E0305 04:39:39.749986 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:07.518584 2604473 out.go:239] Mar 05 04:39:46 old-k8s-version-929157 kubelet[662]: E0305 04:39:46.750507 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:07.518596 2604473 out.go:239] Mar 05 04:39:51 old-k8s-version-929157 kubelet[662]: E0305 04:39:51.749596 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:07.518602 2604473 out.go:239] Mar 05 04:39:58 old-k8s-version-929157 kubelet[662]: E0305 04:39:58.753750 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:07.518614 2604473 out.go:239] Mar 05 04:40:03 old-k8s-version-929157 kubelet[662]: E0305 04:40:03.749667 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
I0305 04:40:07.518622 2604473 out.go:304] Setting ErrFile to fd 2...
I0305 04:40:07.518630 2604473 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0305 04:40:06.242974 2614612 out.go:204] * Creating docker container (CPUs=2, Memory=2200MB) ...
I0305 04:40:06.243248 2614612 start.go:159] libmachine.API.Create for "embed-certs-046985" (driver="docker")
I0305 04:40:06.243280 2614612 client.go:168] LocalClient.Create starting
I0305 04:40:06.243354 2614612 main.go:141] libmachine: Reading certificate data from /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca.pem
I0305 04:40:06.243394 2614612 main.go:141] libmachine: Decoding PEM data...
I0305 04:40:06.243407 2614612 main.go:141] libmachine: Parsing certificate...
I0305 04:40:06.243517 2614612 main.go:141] libmachine: Reading certificate data from /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/cert.pem
I0305 04:40:06.243543 2614612 main.go:141] libmachine: Decoding PEM data...
I0305 04:40:06.243554 2614612 main.go:141] libmachine: Parsing certificate...
I0305 04:40:06.243967 2614612 cli_runner.go:164] Run: docker network inspect embed-certs-046985 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
W0305 04:40:06.265268 2614612 cli_runner.go:211] docker network inspect embed-certs-046985 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1
I0305 04:40:06.265349 2614612 network_create.go:281] running [docker network inspect embed-certs-046985] to gather additional debugging logs...
I0305 04:40:06.265368 2614612 cli_runner.go:164] Run: docker network inspect embed-certs-046985
W0305 04:40:06.291855 2614612 cli_runner.go:211] docker network inspect embed-certs-046985 returned with exit code 1
I0305 04:40:06.291905 2614612 network_create.go:284] error running [docker network inspect embed-certs-046985]: docker network inspect embed-certs-046985: exit status 1
stdout:
[]
stderr:
Error response from daemon: network embed-certs-046985 not found
I0305 04:40:06.291918 2614612 network_create.go:286] output of [docker network inspect embed-certs-046985]: -- stdout --
[]
-- /stdout --
** stderr **
Error response from daemon: network embed-certs-046985 not found
** /stderr **
I0305 04:40:06.292020 2614612 cli_runner.go:164] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0305 04:40:06.315341 2614612 network.go:211] skipping subnet 192.168.49.0/24 that is taken: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName:br-f2ffdcda17e5 IfaceIPv4:192.168.49.1 IfaceMTU:1500 IfaceMAC:02:42:0c:90:00:14} reservation:<nil>}
I0305 04:40:06.315937 2614612 network.go:211] skipping subnet 192.168.58.0/24 that is taken: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 IsPrivate:true Interface:{IfaceName:br-9366e228c701 IfaceIPv4:192.168.58.1 IfaceMTU:1500 IfaceMAC:02:42:ef:2e:19:d9} reservation:<nil>}
I0305 04:40:06.316390 2614612 network.go:211] skipping subnet 192.168.67.0/24 that is taken: &{IP:192.168.67.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.67.0/24 Gateway:192.168.67.1 ClientMin:192.168.67.2 ClientMax:192.168.67.254 Broadcast:192.168.67.255 IsPrivate:true Interface:{IfaceName:br-e809f76046fe IfaceIPv4:192.168.67.1 IfaceMTU:1500 IfaceMAC:02:42:67:25:e2:a8} reservation:<nil>}
I0305 04:40:06.316830 2614612 network.go:211] skipping subnet 192.168.76.0/24 that is taken: &{IP:192.168.76.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.76.0/24 Gateway:192.168.76.1 ClientMin:192.168.76.2 ClientMax:192.168.76.254 Broadcast:192.168.76.255 IsPrivate:true Interface:{IfaceName:br-6ec5655ad76a IfaceIPv4:192.168.76.1 IfaceMTU:1500 IfaceMAC:02:42:a1:94:b3:ac} reservation:<nil>}
I0305 04:40:06.317432 2614612 network.go:206] using free private subnet 192.168.85.0/24: &{IP:192.168.85.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.85.0/24 Gateway:192.168.85.1 ClientMin:192.168.85.2 ClientMax:192.168.85.254 Broadcast:192.168.85.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0x40025ef750}
I0305 04:40:06.317501 2614612 network_create.go:124] attempt to create docker network embed-certs-046985 192.168.85.0/24 with gateway 192.168.85.1 and MTU of 1500 ...
I0305 04:40:06.317599 2614612 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.85.0/24 --gateway=192.168.85.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=1500 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=embed-certs-046985 embed-certs-046985
I0305 04:40:06.408978 2614612 network_create.go:108] docker network embed-certs-046985 192.168.85.0/24 created
I0305 04:40:06.409005 2614612 kic.go:121] calculated static IP "192.168.85.2" for the "embed-certs-046985" container
I0305 04:40:06.409090 2614612 cli_runner.go:164] Run: docker ps -a --format {{.Names}}
I0305 04:40:06.448468 2614612 cli_runner.go:164] Run: docker volume create embed-certs-046985 --label name.minikube.sigs.k8s.io=embed-certs-046985 --label created_by.minikube.sigs.k8s.io=true
I0305 04:40:06.466607 2614612 oci.go:103] Successfully created a docker volume embed-certs-046985
I0305 04:40:06.466697 2614612 cli_runner.go:164] Run: docker run --rm --name embed-certs-046985-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=embed-certs-046985 --entrypoint /usr/bin/test -v embed-certs-046985:/var gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 -d /var/lib
I0305 04:40:07.276608 2614612 oci.go:107] Successfully prepared a docker volume embed-certs-046985
I0305 04:40:07.276665 2614612 preload.go:132] Checking if preload exists for k8s version v1.28.4 and runtime containerd
I0305 04:40:07.276685 2614612 kic.go:194] Starting extracting preloaded images to volume ...
I0305 04:40:07.276801 2614612 cli_runner.go:164] Run: docker run --rm --entrypoint /usr/bin/tar -v /home/jenkins/minikube-integration/17909-2401808/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.28.4-containerd-overlay2-arm64.tar.lz4:/preloaded.tar:ro -v embed-certs-046985:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 -I lz4 -xf /preloaded.tar -C /extractDir
I0305 04:40:12.282322 2614612 cli_runner.go:217] Completed: docker run --rm --entrypoint /usr/bin/tar -v /home/jenkins/minikube-integration/17909-2401808/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.28.4-containerd-overlay2-arm64.tar.lz4:/preloaded.tar:ro -v embed-certs-046985:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 -I lz4 -xf /preloaded.tar -C /extractDir: (5.005479252s)
I0305 04:40:12.282359 2614612 kic.go:203] duration metric: took 5.005670452s to extract preloaded images to volume ...
W0305 04:40:12.282520 2614612 cgroups_linux.go:77] Your kernel does not support swap limit capabilities or the cgroup is not mounted.
I0305 04:40:12.282643 2614612 cli_runner.go:164] Run: docker info --format "'{{json .SecurityOptions}}'"
I0305 04:40:12.347150 2614612 cli_runner.go:164] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname embed-certs-046985 --name embed-certs-046985 --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=embed-certs-046985 --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=embed-certs-046985 --network embed-certs-046985 --ip 192.168.85.2 --volume embed-certs-046985:/var --security-opt apparmor=unconfined --memory=2200mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08
I0305 04:40:12.724175 2614612 cli_runner.go:164] Run: docker container inspect embed-certs-046985 --format={{.State.Running}}
I0305 04:40:12.745615 2614612 cli_runner.go:164] Run: docker container inspect embed-certs-046985 --format={{.State.Status}}
I0305 04:40:12.773685 2614612 cli_runner.go:164] Run: docker exec embed-certs-046985 stat /var/lib/dpkg/alternatives/iptables
I0305 04:40:12.845547 2614612 oci.go:144] the created container "embed-certs-046985" has a running status.
I0305 04:40:12.845586 2614612 kic.go:225] Creating ssh key for kic: /home/jenkins/minikube-integration/17909-2401808/.minikube/machines/embed-certs-046985/id_rsa...
I0305 04:40:13.390740 2614612 kic_runner.go:191] docker (temp): /home/jenkins/minikube-integration/17909-2401808/.minikube/machines/embed-certs-046985/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0305 04:40:13.418478 2614612 cli_runner.go:164] Run: docker container inspect embed-certs-046985 --format={{.State.Status}}
I0305 04:40:13.452095 2614612 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0305 04:40:13.452115 2614612 kic_runner.go:114] Args: [docker exec --privileged embed-certs-046985 chown docker:docker /home/docker/.ssh/authorized_keys]
I0305 04:40:13.535189 2614612 cli_runner.go:164] Run: docker container inspect embed-certs-046985 --format={{.State.Status}}
I0305 04:40:13.563507 2614612 machine.go:94] provisionDockerMachine start ...
I0305 04:40:13.563608 2614612 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-046985
I0305 04:40:13.602583 2614612 main.go:141] libmachine: Using SSH client type: native
I0305 04:40:13.602871 2614612 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3e1c40] 0x3e44a0 <nil> [] 0s} 127.0.0.1 35362 <nil> <nil>}
I0305 04:40:13.602885 2614612 main.go:141] libmachine: About to run SSH command:
hostname
I0305 04:40:13.755307 2614612 main.go:141] libmachine: SSH cmd err, output: <nil>: embed-certs-046985
I0305 04:40:13.755397 2614612 ubuntu.go:169] provisioning hostname "embed-certs-046985"
I0305 04:40:13.755545 2614612 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-046985
I0305 04:40:13.777113 2614612 main.go:141] libmachine: Using SSH client type: native
I0305 04:40:13.777353 2614612 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3e1c40] 0x3e44a0 <nil> [] 0s} 127.0.0.1 35362 <nil> <nil>}
I0305 04:40:13.777364 2614612 main.go:141] libmachine: About to run SSH command:
sudo hostname embed-certs-046985 && echo "embed-certs-046985" | sudo tee /etc/hostname
I0305 04:40:13.935142 2614612 main.go:141] libmachine: SSH cmd err, output: <nil>: embed-certs-046985
I0305 04:40:13.935220 2614612 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-046985
I0305 04:40:13.955345 2614612 main.go:141] libmachine: Using SSH client type: native
I0305 04:40:13.955676 2614612 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3e1c40] 0x3e44a0 <nil> [] 0s} 127.0.0.1 35362 <nil> <nil>}
I0305 04:40:13.955703 2614612 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sembed-certs-046985' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 embed-certs-046985/g' /etc/hosts;
else
echo '127.0.1.1 embed-certs-046985' | sudo tee -a /etc/hosts;
fi
fi
I0305 04:40:14.095920 2614612 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0305 04:40:14.095946 2614612 ubuntu.go:175] set auth options {CertDir:/home/jenkins/minikube-integration/17909-2401808/.minikube CaCertPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca.pem CaPrivateKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/server.pem ServerKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/server-key.pem ClientKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/home/jenkins/minikube-integration/17909-2401808/.minikube}
I0305 04:40:14.095964 2614612 ubuntu.go:177] setting up certificates
I0305 04:40:14.095973 2614612 provision.go:84] configureAuth start
I0305 04:40:14.096031 2614612 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" embed-certs-046985
I0305 04:40:14.124379 2614612 provision.go:143] copyHostCerts
I0305 04:40:14.124448 2614612 exec_runner.go:144] found /home/jenkins/minikube-integration/17909-2401808/.minikube/ca.pem, removing ...
I0305 04:40:14.124462 2614612 exec_runner.go:203] rm: /home/jenkins/minikube-integration/17909-2401808/.minikube/ca.pem
I0305 04:40:14.124547 2614612 exec_runner.go:151] cp: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca.pem --> /home/jenkins/minikube-integration/17909-2401808/.minikube/ca.pem (1082 bytes)
I0305 04:40:14.124649 2614612 exec_runner.go:144] found /home/jenkins/minikube-integration/17909-2401808/.minikube/cert.pem, removing ...
I0305 04:40:14.124660 2614612 exec_runner.go:203] rm: /home/jenkins/minikube-integration/17909-2401808/.minikube/cert.pem
I0305 04:40:14.124688 2614612 exec_runner.go:151] cp: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/cert.pem --> /home/jenkins/minikube-integration/17909-2401808/.minikube/cert.pem (1123 bytes)
I0305 04:40:14.124755 2614612 exec_runner.go:144] found /home/jenkins/minikube-integration/17909-2401808/.minikube/key.pem, removing ...
I0305 04:40:14.124764 2614612 exec_runner.go:203] rm: /home/jenkins/minikube-integration/17909-2401808/.minikube/key.pem
I0305 04:40:14.124789 2614612 exec_runner.go:151] cp: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/key.pem --> /home/jenkins/minikube-integration/17909-2401808/.minikube/key.pem (1679 bytes)
I0305 04:40:14.124850 2614612 provision.go:117] generating server cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/machines/server.pem ca-key=/home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca.pem private-key=/home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca-key.pem org=jenkins.embed-certs-046985 san=[127.0.0.1 192.168.85.2 embed-certs-046985 localhost minikube]
I0305 04:40:14.416367 2614612 provision.go:177] copyRemoteCerts
I0305 04:40:14.416445 2614612 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0305 04:40:14.416491 2614612 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-046985
I0305 04:40:14.432347 2614612 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:35362 SSHKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/embed-certs-046985/id_rsa Username:docker}
I0305 04:40:14.525302 2614612 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1082 bytes)
I0305 04:40:14.550811 2614612 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/machines/server.pem --> /etc/docker/server.pem (1224 bytes)
I0305 04:40:14.579039 2614612 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1679 bytes)
I0305 04:40:14.608418 2614612 provision.go:87] duration metric: took 512.431915ms to configureAuth
I0305 04:40:14.608446 2614612 ubuntu.go:193] setting minikube options for container-runtime
I0305 04:40:14.608629 2614612 config.go:182] Loaded profile config "embed-certs-046985": Driver=docker, ContainerRuntime=containerd, KubernetesVersion=v1.28.4
I0305 04:40:14.608642 2614612 machine.go:97] duration metric: took 1.045112983s to provisionDockerMachine
I0305 04:40:14.608649 2614612 client.go:171] duration metric: took 8.365364345s to LocalClient.Create
I0305 04:40:14.608666 2614612 start.go:167] duration metric: took 8.36541926s to libmachine.API.Create "embed-certs-046985"
I0305 04:40:14.608673 2614612 start.go:293] postStartSetup for "embed-certs-046985" (driver="docker")
I0305 04:40:14.608683 2614612 start.go:322] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0305 04:40:14.608729 2614612 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0305 04:40:14.608777 2614612 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-046985
I0305 04:40:14.625163 2614612 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:35362 SSHKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/embed-certs-046985/id_rsa Username:docker}
I0305 04:40:14.720855 2614612 ssh_runner.go:195] Run: cat /etc/os-release
I0305 04:40:14.724354 2614612 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0305 04:40:14.724389 2614612 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0305 04:40:14.724399 2614612 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0305 04:40:14.724407 2614612 info.go:137] Remote host: Ubuntu 22.04.3 LTS
I0305 04:40:14.724418 2614612 filesync.go:126] Scanning /home/jenkins/minikube-integration/17909-2401808/.minikube/addons for local assets ...
I0305 04:40:14.724479 2614612 filesync.go:126] Scanning /home/jenkins/minikube-integration/17909-2401808/.minikube/files for local assets ...
I0305 04:40:14.724573 2614612 filesync.go:149] local asset: /home/jenkins/minikube-integration/17909-2401808/.minikube/files/etc/ssl/certs/24071502.pem -> 24071502.pem in /etc/ssl/certs
I0305 04:40:14.724713 2614612 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0305 04:40:14.733424 2614612 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/files/etc/ssl/certs/24071502.pem --> /etc/ssl/certs/24071502.pem (1708 bytes)
I0305 04:40:14.762499 2614612 start.go:296] duration metric: took 153.811432ms for postStartSetup
I0305 04:40:14.762915 2614612 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" embed-certs-046985
I0305 04:40:14.779782 2614612 profile.go:142] Saving config to /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/config.json ...
I0305 04:40:14.780057 2614612 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0305 04:40:14.780112 2614612 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-046985
I0305 04:40:14.795528 2614612 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:35362 SSHKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/embed-certs-046985/id_rsa Username:docker}
I0305 04:40:14.888322 2614612 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0305 04:40:14.892916 2614612 start.go:128] duration metric: took 8.653856922s to createHost
I0305 04:40:14.892942 2614612 start.go:83] releasing machines lock for "embed-certs-046985", held for 8.654031219s
I0305 04:40:14.893011 2614612 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" embed-certs-046985
I0305 04:40:14.909677 2614612 ssh_runner.go:195] Run: cat /version.json
I0305 04:40:14.909739 2614612 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-046985
I0305 04:40:14.910019 2614612 ssh_runner.go:195] Run: curl -sS -m 2 https://registry.k8s.io/
I0305 04:40:14.910080 2614612 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-046985
I0305 04:40:14.936694 2614612 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:35362 SSHKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/embed-certs-046985/id_rsa Username:docker}
I0305 04:40:14.939729 2614612 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:35362 SSHKeyPath:/home/jenkins/minikube-integration/17909-2401808/.minikube/machines/embed-certs-046985/id_rsa Username:docker}
I0305 04:40:15.032441 2614612 ssh_runner.go:195] Run: systemctl --version
I0305 04:40:15.151803 2614612 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
I0305 04:40:15.156852 2614612 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *loopback.conf* -not -name *.mk_disabled -exec sh -c "grep -q loopback {} && ( grep -q name {} || sudo sed -i '/"type": "loopback"/i \ \ \ \ "name": "loopback",' {} ) && sudo sed -i 's|"cniVersion": ".*"|"cniVersion": "1.0.0"|g' {}" ;
I0305 04:40:15.189287 2614612 cni.go:230] loopback cni configuration patched: "/etc/cni/net.d/*loopback.conf*" found
I0305 04:40:15.189439 2614612 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f ( ( -name *bridge* -or -name *podman* ) -and -not -name *.mk_disabled ) -printf "%!p(MISSING), " -exec sh -c "sudo mv {} {}.mk_disabled" ;
I0305 04:40:15.223234 2614612 cni.go:262] disabled [/etc/cni/net.d/87-podman-bridge.conflist, /etc/cni/net.d/100-crio-bridge.conf] bridge cni config(s)
I0305 04:40:15.223276 2614612 start.go:494] detecting cgroup driver to use...
I0305 04:40:15.223335 2614612 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0305 04:40:15.223394 2614612 ssh_runner.go:195] Run: sudo systemctl stop -f crio
I0305 04:40:15.236834 2614612 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0305 04:40:15.249120 2614612 docker.go:217] disabling cri-docker service (if available) ...
I0305 04:40:15.249211 2614612 ssh_runner.go:195] Run: sudo systemctl stop -f cri-docker.socket
I0305 04:40:15.264480 2614612 ssh_runner.go:195] Run: sudo systemctl stop -f cri-docker.service
I0305 04:40:15.280108 2614612 ssh_runner.go:195] Run: sudo systemctl disable cri-docker.socket
I0305 04:40:15.373562 2614612 ssh_runner.go:195] Run: sudo systemctl mask cri-docker.service
I0305 04:40:15.475946 2614612 docker.go:233] disabling docker service ...
I0305 04:40:15.476009 2614612 ssh_runner.go:195] Run: sudo systemctl stop -f docker.socket
I0305 04:40:15.500303 2614612 ssh_runner.go:195] Run: sudo systemctl stop -f docker.service
I0305 04:40:15.512911 2614612 ssh_runner.go:195] Run: sudo systemctl disable docker.socket
I0305 04:40:15.617084 2614612 ssh_runner.go:195] Run: sudo systemctl mask docker.service
I0305 04:40:15.713298 2614612 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service docker
I0305 04:40:15.726345 2614612 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %!s(MISSING) "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0305 04:40:15.743604 2614612 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.9"|' /etc/containerd/config.toml"
I0305 04:40:15.754685 2614612 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I0305 04:40:15.764844 2614612 containerd.go:146] configuring containerd to use "cgroupfs" as cgroup driver...
I0305 04:40:15.764912 2614612 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = false|g' /etc/containerd/config.toml"
I0305 04:40:15.776885 2614612 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0305 04:40:15.786675 2614612 ssh_runner.go:195] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I0305 04:40:15.796821 2614612 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0305 04:40:15.806977 2614612 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0305 04:40:15.816629 2614612 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I0305 04:40:15.827353 2614612 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0305 04:40:15.836422 2614612 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0305 04:40:15.845325 2614612 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0305 04:40:15.939603 2614612 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0305 04:40:16.083762 2614612 start.go:541] Will wait 60s for socket path /run/containerd/containerd.sock
I0305 04:40:16.083837 2614612 ssh_runner.go:195] Run: stat /run/containerd/containerd.sock
I0305 04:40:16.087849 2614612 start.go:562] Will wait 60s for crictl version
I0305 04:40:16.087927 2614612 ssh_runner.go:195] Run: which crictl
I0305 04:40:16.092154 2614612 ssh_runner.go:195] Run: sudo /usr/bin/crictl version
I0305 04:40:16.132689 2614612 start.go:578] Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: 1.6.28
RuntimeApiVersion: v1
I0305 04:40:16.132760 2614612 ssh_runner.go:195] Run: containerd --version
I0305 04:40:16.157046 2614612 ssh_runner.go:195] Run: containerd --version
I0305 04:40:16.183778 2614612 out.go:177] * Preparing Kubernetes v1.28.4 on containerd 1.6.28 ...
I0305 04:40:16.185664 2614612 cli_runner.go:164] Run: docker network inspect embed-certs-046985 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0305 04:40:16.201199 2614612 ssh_runner.go:195] Run: grep 192.168.85.1 host.minikube.internal$ /etc/hosts
I0305 04:40:16.204893 2614612 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.85.1 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0305 04:40:16.216103 2614612 kubeadm.go:877] updating cluster {Name:embed-certs-046985 KeepContext:false EmbedCerts:true MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.28.4 ClusterName:embed-certs-046985 Namespace:default APIServerHAVIP: APIServerName:minikubeCA AP
IServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.85.2 Port:8443 KubernetesVersion:v1.28.4 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:
false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s} ...
I0305 04:40:16.216233 2614612 preload.go:132] Checking if preload exists for k8s version v1.28.4 and runtime containerd
I0305 04:40:16.216298 2614612 ssh_runner.go:195] Run: sudo crictl images --output json
I0305 04:40:16.253170 2614612 containerd.go:612] all images are preloaded for containerd runtime.
I0305 04:40:16.253195 2614612 containerd.go:519] Images already preloaded, skipping extraction
I0305 04:40:16.253261 2614612 ssh_runner.go:195] Run: sudo crictl images --output json
I0305 04:40:16.291153 2614612 containerd.go:612] all images are preloaded for containerd runtime.
I0305 04:40:16.291177 2614612 cache_images.go:84] Images are preloaded, skipping loading
I0305 04:40:16.291186 2614612 kubeadm.go:928] updating node { 192.168.85.2 8443 v1.28.4 containerd true true} ...
I0305 04:40:16.291279 2614612 kubeadm.go:940] kubelet [Unit]
Wants=containerd.service
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.28.4/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --hostname-override=embed-certs-046985 --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.85.2
[Install]
config:
{KubernetesVersion:v1.28.4 ClusterName:embed-certs-046985 Namespace:default APIServerHAVIP: APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:}
I0305 04:40:16.291349 2614612 ssh_runner.go:195] Run: sudo crictl info
I0305 04:40:16.338322 2614612 cni.go:84] Creating CNI manager for ""
I0305 04:40:16.338344 2614612 cni.go:143] "docker" driver + "containerd" runtime found, recommending kindnet
I0305 04:40:16.338354 2614612 kubeadm.go:84] Using pod CIDR: 10.244.0.0/16
I0305 04:40:16.338377 2614612 kubeadm.go:181] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.85.2 APIServerPort:8443 KubernetesVersion:v1.28.4 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:embed-certs-046985 NodeName:embed-certs-046985 DNSDomain:cluster.local CRISocket:/run/containerd/containerd.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.85.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.85.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticPo
dPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[containerRuntimeEndpoint:unix:///run/containerd/containerd.sock hairpinMode:hairpin-veth runtimeRequestTimeout:15m] PrependCriSocketUnix:true}
I0305 04:40:16.338510 2614612 kubeadm.go:187] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.85.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: unix:///run/containerd/containerd.sock
name: "embed-certs-046985"
kubeletExtraArgs:
node-ip: 192.168.85.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.85.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: mk
controlPlaneEndpoint: control-plane.minikube.internal:8443
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
proxy-refresh-interval: "70000"
kubernetesVersion: v1.28.4
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
containerRuntimeEndpoint: unix:///run/containerd/containerd.sock
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%!"(MISSING)
nodefs.inodesFree: "0%!"(MISSING)
imagefs.available: "0%!"(MISSING)
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0305 04:40:16.338578 2614612 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.28.4
I0305 04:40:16.349009 2614612 binaries.go:44] Found k8s binaries, skipping transfer
I0305 04:40:16.349079 2614612 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0305 04:40:16.358464 2614612 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (322 bytes)
I0305 04:40:16.378992 2614612 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0305 04:40:16.398349 2614612 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2172 bytes)
I0305 04:40:16.417256 2614612 ssh_runner.go:195] Run: grep 192.168.85.2 control-plane.minikube.internal$ /etc/hosts
I0305 04:40:16.420938 2614612 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.85.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0305 04:40:16.432253 2614612 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0305 04:40:16.518447 2614612 ssh_runner.go:195] Run: sudo systemctl start kubelet
I0305 04:40:16.536064 2614612 certs.go:68] Setting up /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985 for IP: 192.168.85.2
I0305 04:40:16.536137 2614612 certs.go:194] generating shared ca certs ...
I0305 04:40:16.536167 2614612 certs.go:226] acquiring lock for ca certs: {Name:mkcd340b494959143cea31c1bd7ad2cead6bae3a Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0305 04:40:16.536337 2614612 certs.go:235] skipping valid "minikubeCA" ca cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/ca.key
I0305 04:40:16.536407 2614612 certs.go:235] skipping valid "proxyClientCA" ca cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/proxy-client-ca.key
I0305 04:40:16.536429 2614612 certs.go:256] generating profile certs ...
I0305 04:40:16.536526 2614612 certs.go:363] generating signed profile cert for "minikube-user": /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/client.key
I0305 04:40:16.536569 2614612 crypto.go:68] Generating cert /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/client.crt with IP's: []
I0305 04:40:16.664481 2614612 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/client.crt ...
I0305 04:40:16.664513 2614612 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/client.crt: {Name:mkbb4f0773f0ec5b1dab39cd1300785a041b915d Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0305 04:40:16.664738 2614612 crypto.go:164] Writing key to /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/client.key ...
I0305 04:40:16.664755 2614612 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/client.key: {Name:mk81586aa8e9d103966e0b4c53095b8a9be48d36 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0305 04:40:16.665348 2614612 certs.go:363] generating signed profile cert for "minikube": /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/apiserver.key.82ca6ed0
I0305 04:40:16.665371 2614612 crypto.go:68] Generating cert /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/apiserver.crt.82ca6ed0 with IP's: [10.96.0.1 127.0.0.1 10.0.0.1 192.168.85.2]
I0305 04:40:16.806548 2614612 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/apiserver.crt.82ca6ed0 ...
I0305 04:40:16.806581 2614612 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/apiserver.crt.82ca6ed0: {Name:mkdfd4df51b9877263de9663d5d92a7e4892eb1d Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0305 04:40:16.806814 2614612 crypto.go:164] Writing key to /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/apiserver.key.82ca6ed0 ...
I0305 04:40:16.806831 2614612 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/apiserver.key.82ca6ed0: {Name:mkc6c7a351097d63567f4c8f00563c9040c44488 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0305 04:40:16.807600 2614612 certs.go:381] copying /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/apiserver.crt.82ca6ed0 -> /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/apiserver.crt
I0305 04:40:16.807738 2614612 certs.go:385] copying /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/apiserver.key.82ca6ed0 -> /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/apiserver.key
I0305 04:40:16.807812 2614612 certs.go:363] generating signed profile cert for "aggregator": /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/proxy-client.key
I0305 04:40:16.807844 2614612 crypto.go:68] Generating cert /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/proxy-client.crt with IP's: []
I0305 04:40:16.971796 2614612 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/proxy-client.crt ...
I0305 04:40:16.971829 2614612 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/proxy-client.crt: {Name:mkf8ff706762187644fb48affb2500fe5a61cc43 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0305 04:40:16.972431 2614612 crypto.go:164] Writing key to /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/proxy-client.key ...
I0305 04:40:16.972450 2614612 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/proxy-client.key: {Name:mk1f939c9cbcb38886527981b3e3361917d4dc3b Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0305 04:40:16.972659 2614612 certs.go:484] found cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/2407150.pem (1338 bytes)
W0305 04:40:16.972714 2614612 certs.go:480] ignoring /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/2407150_empty.pem, impossibly tiny 0 bytes
I0305 04:40:16.972729 2614612 certs.go:484] found cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca-key.pem (1675 bytes)
I0305 04:40:16.972754 2614612 certs.go:484] found cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/ca.pem (1082 bytes)
I0305 04:40:16.972782 2614612 certs.go:484] found cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/cert.pem (1123 bytes)
I0305 04:40:16.972809 2614612 certs.go:484] found cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/key.pem (1679 bytes)
I0305 04:40:16.972855 2614612 certs.go:484] found cert: /home/jenkins/minikube-integration/17909-2401808/.minikube/files/etc/ssl/certs/24071502.pem (1708 bytes)
I0305 04:40:16.973498 2614612 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0305 04:40:16.999814 2614612 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1679 bytes)
I0305 04:40:17.029776 2614612 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0305 04:40:17.056985 2614612 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1675 bytes)
I0305 04:40:17.084221 2614612 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1428 bytes)
I0305 04:40:17.112161 2614612 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1675 bytes)
I0305 04:40:17.140035 2614612 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0305 04:40:17.171558 2614612 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/profiles/embed-certs-046985/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1679 bytes)
I0305 04:40:17.202396 2614612 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0305 04:40:17.228766 2614612 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/certs/2407150.pem --> /usr/share/ca-certificates/2407150.pem (1338 bytes)
I0305 04:40:17.256615 2614612 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17909-2401808/.minikube/files/etc/ssl/certs/24071502.pem --> /usr/share/ca-certificates/24071502.pem (1708 bytes)
I0305 04:40:17.282998 2614612 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0305 04:40:17.301609 2614612 ssh_runner.go:195] Run: openssl version
I0305 04:40:17.307500 2614612 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0305 04:40:17.317433 2614612 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0305 04:40:17.321417 2614612 certs.go:528] hashing: -rw-r--r-- 1 root root 1111 Mar 5 03:48 /usr/share/ca-certificates/minikubeCA.pem
I0305 04:40:17.321482 2614612 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0305 04:40:17.328805 2614612 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0305 04:40:17.338156 2614612 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/2407150.pem && ln -fs /usr/share/ca-certificates/2407150.pem /etc/ssl/certs/2407150.pem"
I0305 04:40:17.347851 2614612 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/2407150.pem
I0305 04:40:17.351572 2614612 certs.go:528] hashing: -rw-r--r-- 1 root root 1338 Mar 5 03:53 /usr/share/ca-certificates/2407150.pem
I0305 04:40:17.351658 2614612 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/2407150.pem
I0305 04:40:17.358628 2614612 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/2407150.pem /etc/ssl/certs/51391683.0"
I0305 04:40:17.368377 2614612 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/24071502.pem && ln -fs /usr/share/ca-certificates/24071502.pem /etc/ssl/certs/24071502.pem"
I0305 04:40:17.379222 2614612 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/24071502.pem
I0305 04:40:17.383157 2614612 certs.go:528] hashing: -rw-r--r-- 1 root root 1708 Mar 5 03:53 /usr/share/ca-certificates/24071502.pem
I0305 04:40:17.383229 2614612 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/24071502.pem
I0305 04:40:17.390844 2614612 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/24071502.pem /etc/ssl/certs/3ec20f2e.0"
I0305 04:40:17.401150 2614612 ssh_runner.go:195] Run: stat /var/lib/minikube/certs/apiserver-kubelet-client.crt
I0305 04:40:17.405146 2614612 certs.go:399] 'apiserver-kubelet-client' cert doesn't exist, likely first start: stat /var/lib/minikube/certs/apiserver-kubelet-client.crt: Process exited with status 1
stdout:
stderr:
stat: cannot statx '/var/lib/minikube/certs/apiserver-kubelet-client.crt': No such file or directory
I0305 04:40:17.405205 2614612 kubeadm.go:391] StartCluster: {Name:embed-certs-046985 KeepContext:false EmbedCerts:true MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708944392-18244@sha256:8610dac8144c3f59a6cf50871eb10395cea122e148262744231a04c396033b08 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.28.4 ClusterName:embed-certs-046985 Namespace:default APIServerHAVIP: APIServerName:minikubeCA APISe
rverNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.85.2 Port:8443 KubernetesVersion:v1.28.4 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:fal
se CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0305 04:40:17.405312 2614612 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:paused Name: Namespaces:[kube-system]}
I0305 04:40:17.405381 2614612 ssh_runner.go:195] Run: sudo -s eval "crictl ps -a --quiet --label io.kubernetes.pod.namespace=kube-system"
I0305 04:40:17.457058 2614612 cri.go:89] found id: ""
I0305 04:40:17.457132 2614612 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0305 04:40:17.466602 2614612 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0305 04:40:17.476306 2614612 kubeadm.go:213] ignoring SystemVerification for kubeadm because of docker driver
I0305 04:40:17.476372 2614612 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0305 04:40:17.485599 2614612 kubeadm.go:154] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0305 04:40:17.485620 2614612 kubeadm.go:156] found existing configuration files:
I0305 04:40:17.485672 2614612 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/admin.conf
I0305 04:40:17.494629 2614612 kubeadm.go:162] "https://control-plane.minikube.internal:8443" may not be in /etc/kubernetes/admin.conf - will remove: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/admin.conf: Process exited with status 2
stdout:
stderr:
grep: /etc/kubernetes/admin.conf: No such file or directory
I0305 04:40:17.494715 2614612 ssh_runner.go:195] Run: sudo rm -f /etc/kubernetes/admin.conf
I0305 04:40:17.503333 2614612 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/kubelet.conf
I0305 04:40:17.512621 2614612 kubeadm.go:162] "https://control-plane.minikube.internal:8443" may not be in /etc/kubernetes/kubelet.conf - will remove: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/kubelet.conf: Process exited with status 2
stdout:
stderr:
grep: /etc/kubernetes/kubelet.conf: No such file or directory
I0305 04:40:17.512714 2614612 ssh_runner.go:195] Run: sudo rm -f /etc/kubernetes/kubelet.conf
I0305 04:40:17.523943 2614612 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/controller-manager.conf
I0305 04:40:17.537145 2614612 kubeadm.go:162] "https://control-plane.minikube.internal:8443" may not be in /etc/kubernetes/controller-manager.conf - will remove: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/controller-manager.conf: Process exited with status 2
stdout:
stderr:
grep: /etc/kubernetes/controller-manager.conf: No such file or directory
I0305 04:40:17.537262 2614612 ssh_runner.go:195] Run: sudo rm -f /etc/kubernetes/controller-manager.conf
I0305 04:40:17.548645 2614612 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/scheduler.conf
I0305 04:40:17.560835 2614612 kubeadm.go:162] "https://control-plane.minikube.internal:8443" may not be in /etc/kubernetes/scheduler.conf - will remove: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
grep: /etc/kubernetes/scheduler.conf: No such file or directory
I0305 04:40:17.560952 2614612 ssh_runner.go:195] Run: sudo rm -f /etc/kubernetes/scheduler.conf
I0305 04:40:17.573026 2614612 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.28.4:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0305 04:40:17.642271 2614612 kubeadm.go:309] [init] Using Kubernetes version: v1.28.4
I0305 04:40:17.642397 2614612 kubeadm.go:309] [preflight] Running pre-flight checks
I0305 04:40:17.702863 2614612 kubeadm.go:309] [preflight] The system verification failed. Printing the output from the verification:
I0305 04:40:17.702957 2614612 kubeadm.go:309] [0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1055-aws[0m
I0305 04:40:17.703009 2614612 kubeadm.go:309] [0;37mOS[0m: [0;32mLinux[0m
I0305 04:40:17.703070 2614612 kubeadm.go:309] [0;37mCGROUPS_CPU[0m: [0;32menabled[0m
I0305 04:40:17.703132 2614612 kubeadm.go:309] [0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
I0305 04:40:17.703182 2614612 kubeadm.go:309] [0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
I0305 04:40:17.703246 2614612 kubeadm.go:309] [0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
I0305 04:40:17.703306 2614612 kubeadm.go:309] [0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
I0305 04:40:17.703367 2614612 kubeadm.go:309] [0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
I0305 04:40:17.703422 2614612 kubeadm.go:309] [0;37mCGROUPS_PIDS[0m: [0;32menabled[0m
I0305 04:40:17.703491 2614612 kubeadm.go:309] [0;37mCGROUPS_HUGETLB[0m: [0;32menabled[0m
I0305 04:40:17.703560 2614612 kubeadm.go:309] [0;37mCGROUPS_BLKIO[0m: [0;32menabled[0m
I0305 04:40:17.891697 2614612 kubeadm.go:309] [preflight] Pulling images required for setting up a Kubernetes cluster
I0305 04:40:17.891904 2614612 kubeadm.go:309] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0305 04:40:17.892032 2614612 kubeadm.go:309] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0305 04:40:18.422517 2614612 kubeadm.go:309] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0305 04:40:17.520283 2604473 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0305 04:40:17.538481 2604473 api_server.go:72] duration metric: took 5m48.69883193s to wait for apiserver process to appear ...
I0305 04:40:17.538502 2604473 api_server.go:88] waiting for apiserver healthz status ...
I0305 04:40:17.538536 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0305 04:40:17.538590 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0305 04:40:17.587028 2604473 cri.go:89] found id: "71cb2e24b42a7f8d797580bf0b66535d6aa909eae6127ef3f669c69251df7f53"
I0305 04:40:17.587054 2604473 cri.go:89] found id: "fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259"
I0305 04:40:17.587059 2604473 cri.go:89] found id: ""
I0305 04:40:17.587066 2604473 logs.go:276] 2 containers: [71cb2e24b42a7f8d797580bf0b66535d6aa909eae6127ef3f669c69251df7f53 fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259]
I0305 04:40:17.587121 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.590809 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.594269 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0305 04:40:17.594337 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0305 04:40:17.647126 2604473 cri.go:89] found id: "3dd18badcf58a90e38fe4db8a253226bf84cf241094215382ff4b9cc24ee04a5"
I0305 04:40:17.647155 2604473 cri.go:89] found id: "35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605"
I0305 04:40:17.647160 2604473 cri.go:89] found id: ""
I0305 04:40:17.647167 2604473 logs.go:276] 2 containers: [3dd18badcf58a90e38fe4db8a253226bf84cf241094215382ff4b9cc24ee04a5 35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605]
I0305 04:40:17.647267 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.651142 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.654792 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0305 04:40:17.654857 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0305 04:40:17.714047 2604473 cri.go:89] found id: "e0dbacaa8e30885944ab46b688f9bbefac308f823788712e6c14ecdc1556f552"
I0305 04:40:17.714066 2604473 cri.go:89] found id: "508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99"
I0305 04:40:17.714070 2604473 cri.go:89] found id: ""
I0305 04:40:17.714078 2604473 logs.go:276] 2 containers: [e0dbacaa8e30885944ab46b688f9bbefac308f823788712e6c14ecdc1556f552 508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99]
I0305 04:40:17.714132 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.717844 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.721446 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0305 04:40:17.721558 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0305 04:40:17.836275 2604473 cri.go:89] found id: "a5d43d0d8a5d3f1e42ca4c1b9ec4dd6a812c6454f3858ec238850f1190bce93c"
I0305 04:40:17.836346 2604473 cri.go:89] found id: "27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636"
I0305 04:40:17.836368 2604473 cri.go:89] found id: ""
I0305 04:40:17.836390 2604473 logs.go:276] 2 containers: [a5d43d0d8a5d3f1e42ca4c1b9ec4dd6a812c6454f3858ec238850f1190bce93c 27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636]
I0305 04:40:17.836479 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.843604 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.852424 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0305 04:40:17.852530 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0305 04:40:17.959522 2604473 cri.go:89] found id: "dd4135e2a8d47ee5af337a3314218c302ed97e8f2c878263df59d7b367ea4443"
I0305 04:40:17.959555 2604473 cri.go:89] found id: "acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f"
I0305 04:40:17.959560 2604473 cri.go:89] found id: ""
I0305 04:40:17.959567 2604473 logs.go:276] 2 containers: [dd4135e2a8d47ee5af337a3314218c302ed97e8f2c878263df59d7b367ea4443 acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f]
I0305 04:40:17.959671 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.964637 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:17.969480 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0305 04:40:17.969577 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0305 04:40:18.062276 2604473 cri.go:89] found id: "9a1fcc970baf72fae5fc3eb76833e0232c8f51d7938d48ea284de0940c3a961c"
I0305 04:40:18.062348 2604473 cri.go:89] found id: "e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341"
I0305 04:40:18.062368 2604473 cri.go:89] found id: ""
I0305 04:40:18.062390 2604473 logs.go:276] 2 containers: [9a1fcc970baf72fae5fc3eb76833e0232c8f51d7938d48ea284de0940c3a961c e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341]
I0305 04:40:18.062474 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:18.067015 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:18.071263 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kindnet Namespaces:[]}
I0305 04:40:18.071413 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kindnet
I0305 04:40:18.131770 2604473 cri.go:89] found id: "767fb3c810f4e121fae9120786476ac6029d586fd1a6203f8cf8917dbbaf13cb"
I0305 04:40:18.131845 2604473 cri.go:89] found id: "feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232"
I0305 04:40:18.131867 2604473 cri.go:89] found id: ""
I0305 04:40:18.131892 2604473 logs.go:276] 2 containers: [767fb3c810f4e121fae9120786476ac6029d586fd1a6203f8cf8917dbbaf13cb feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232]
I0305 04:40:18.131974 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:18.136721 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:18.140918 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0305 04:40:18.141043 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0305 04:40:18.192151 2604473 cri.go:89] found id: "4c1bed4d5ba29d0be56313d0a71a5ec6110f707ff84f3f6e34f4028a3d82e073"
I0305 04:40:18.192224 2604473 cri.go:89] found id: ""
I0305 04:40:18.192247 2604473 logs.go:276] 1 containers: [4c1bed4d5ba29d0be56313d0a71a5ec6110f707ff84f3f6e34f4028a3d82e073]
I0305 04:40:18.192329 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:18.197650 2604473 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0305 04:40:18.197770 2604473 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0305 04:40:18.248230 2604473 cri.go:89] found id: "9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff"
I0305 04:40:18.248299 2604473 cri.go:89] found id: "cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7"
I0305 04:40:18.248318 2604473 cri.go:89] found id: ""
I0305 04:40:18.248339 2604473 logs.go:276] 2 containers: [9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7]
I0305 04:40:18.248419 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:18.252542 2604473 ssh_runner.go:195] Run: which crictl
I0305 04:40:18.257770 2604473 logs.go:123] Gathering logs for kube-scheduler [27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636] ...
I0305 04:40:18.257844 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636"
I0305 04:40:18.308816 2604473 logs.go:123] Gathering logs for container status ...
I0305 04:40:18.308886 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0305 04:40:18.377518 2604473 logs.go:123] Gathering logs for describe nodes ...
I0305 04:40:18.377651 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.20.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
I0305 04:40:18.562337 2604473 logs.go:123] Gathering logs for kube-apiserver [71cb2e24b42a7f8d797580bf0b66535d6aa909eae6127ef3f669c69251df7f53] ...
I0305 04:40:18.562481 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 71cb2e24b42a7f8d797580bf0b66535d6aa909eae6127ef3f669c69251df7f53"
I0305 04:40:18.645832 2604473 logs.go:123] Gathering logs for kindnet [767fb3c810f4e121fae9120786476ac6029d586fd1a6203f8cf8917dbbaf13cb] ...
I0305 04:40:18.645909 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 767fb3c810f4e121fae9120786476ac6029d586fd1a6203f8cf8917dbbaf13cb"
I0305 04:40:18.713062 2604473 logs.go:123] Gathering logs for storage-provisioner [cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7] ...
I0305 04:40:18.713194 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7"
I0305 04:40:18.767420 2604473 logs.go:123] Gathering logs for storage-provisioner [9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff] ...
I0305 04:40:18.767505 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff"
I0305 04:40:18.819044 2604473 logs.go:123] Gathering logs for kubelet ...
I0305 04:40:18.819120 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0305 04:40:18.885822 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566136 662 reflector.go:138] object-"kube-system"/"metrics-server-token-z55cz": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "metrics-server-token-z55cz" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.886097 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566265 662 reflector.go:138] object-"kube-system"/"kube-proxy-token-w7vc2": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kube-proxy-token-w7vc2" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.886324 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566376 662 reflector.go:138] object-"kube-system"/"coredns": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "coredns" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.886555 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566444 662 reflector.go:138] object-"kube-system"/"coredns-token-dkvsz": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "coredns-token-dkvsz" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.886787 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566500 662 reflector.go:138] object-"default"/"default-token-xt7xb": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "default-token-xt7xb" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "default": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.887035 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566558 662 reflector.go:138] object-"kube-system"/"storage-provisioner-token-hsz7w": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "storage-provisioner-token-hsz7w" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.887266 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566618 662 reflector.go:138] object-"kube-system"/"kindnet-token-jq4vl": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kindnet-token-jq4vl" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.887501 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:48 old-k8s-version-929157 kubelet[662]: E0305 04:34:48.566677 662 reflector.go:138] object-"kube-system"/"kube-proxy": Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "kube-proxy" is forbidden: User "system:node:old-k8s-version-929157" cannot list resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-929157' and this object
W0305 04:40:18.900193 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:51 old-k8s-version-929157 kubelet[662]: E0305 04:34:51.325447 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:18.900445 2604473 logs.go:138] Found kubelet problem: Mar 05 04:34:51 old-k8s-version-929157 kubelet[662]: E0305 04:34:51.878905 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.903428 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:05 old-k8s-version-929157 kubelet[662]: E0305 04:35:05.758082 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:18.905641 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:14 old-k8s-version-929157 kubelet[662]: E0305 04:35:14.030339 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.905991 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:15 old-k8s-version-929157 kubelet[662]: E0305 04:35:15.047564 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.906521 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:18 old-k8s-version-929157 kubelet[662]: E0305 04:35:18.750259 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.906865 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:22 old-k8s-version-929157 kubelet[662]: E0305 04:35:22.709995 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.907321 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:23 old-k8s-version-929157 kubelet[662]: E0305 04:35:23.063333 662 pod_workers.go:191] Error syncing pod b3db2690-165e-46e7-bef2-d3020d2c11e2 ("storage-provisioner_kube-system(b3db2690-165e-46e7-bef2-d3020d2c11e2)"), skipping: failed to "StartContainer" for "storage-provisioner" with CrashLoopBackOff: "back-off 10s restarting failed container=storage-provisioner pod=storage-provisioner_kube-system(b3db2690-165e-46e7-bef2-d3020d2c11e2)"
W0305 04:40:18.910113 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:30 old-k8s-version-929157 kubelet[662]: E0305 04:35:30.764359 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:18.910881 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:38 old-k8s-version-929157 kubelet[662]: E0305 04:35:38.133208 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.911226 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:42 old-k8s-version-929157 kubelet[662]: E0305 04:35:42.709684 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.911429 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:44 old-k8s-version-929157 kubelet[662]: E0305 04:35:44.750232 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.913473 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:53 old-k8s-version-929157 kubelet[662]: E0305 04:35:53.749653 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.913711 2604473 logs.go:138] Found kubelet problem: Mar 05 04:35:57 old-k8s-version-929157 kubelet[662]: E0305 04:35:57.760380 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.914318 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:08 old-k8s-version-929157 kubelet[662]: E0305 04:36:08.224516 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.914668 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:12 old-k8s-version-929157 kubelet[662]: E0305 04:36:12.710208 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.917119 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:13 old-k8s-version-929157 kubelet[662]: E0305 04:36:13.760058 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:18.917349 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:24 old-k8s-version-929157 kubelet[662]: E0305 04:36:24.750302 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.917692 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:27 old-k8s-version-929157 kubelet[662]: E0305 04:36:27.749613 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.917895 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:35 old-k8s-version-929157 kubelet[662]: E0305 04:36:35.750001 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.918239 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:38 old-k8s-version-929157 kubelet[662]: E0305 04:36:38.749574 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.918443 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:46 old-k8s-version-929157 kubelet[662]: E0305 04:36:46.750380 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.919044 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:51 old-k8s-version-929157 kubelet[662]: E0305 04:36:51.312622 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.919389 2604473 logs.go:138] Found kubelet problem: Mar 05 04:36:52 old-k8s-version-929157 kubelet[662]: E0305 04:36:52.709685 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.919609 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:01 old-k8s-version-929157 kubelet[662]: E0305 04:37:01.749989 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.919955 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:06 old-k8s-version-929157 kubelet[662]: E0305 04:37:06.750188 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.920159 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:12 old-k8s-version-929157 kubelet[662]: E0305 04:37:12.752686 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.920508 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:17 old-k8s-version-929157 kubelet[662]: E0305 04:37:17.749715 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.920714 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:25 old-k8s-version-929157 kubelet[662]: E0305 04:37:25.749931 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.921058 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:28 old-k8s-version-929157 kubelet[662]: E0305 04:37:28.749688 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.921406 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:39 old-k8s-version-929157 kubelet[662]: E0305 04:37:39.749886 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.923883 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:39 old-k8s-version-929157 kubelet[662]: E0305 04:37:39.757581 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
W0305 04:40:18.924233 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:50 old-k8s-version-929157 kubelet[662]: E0305 04:37:50.750942 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.924436 2604473 logs.go:138] Found kubelet problem: Mar 05 04:37:50 old-k8s-version-929157 kubelet[662]: E0305 04:37:50.751501 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.924782 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:02 old-k8s-version-929157 kubelet[662]: E0305 04:38:02.750200 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.924985 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:05 old-k8s-version-929157 kubelet[662]: E0305 04:38:05.750061 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.925588 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:18 old-k8s-version-929157 kubelet[662]: E0305 04:38:18.528409 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.925792 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:19 old-k8s-version-929157 kubelet[662]: E0305 04:38:19.750661 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.926140 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:22 old-k8s-version-929157 kubelet[662]: E0305 04:38:22.709687 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.926351 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:33 old-k8s-version-929157 kubelet[662]: E0305 04:38:33.749936 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.926697 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:36 old-k8s-version-929157 kubelet[662]: E0305 04:38:36.756212 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.926901 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:45 old-k8s-version-929157 kubelet[662]: E0305 04:38:45.749980 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.927248 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:48 old-k8s-version-929157 kubelet[662]: E0305 04:38:48.749677 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.928887 2604473 logs.go:138] Found kubelet problem: Mar 05 04:38:59 old-k8s-version-929157 kubelet[662]: E0305 04:38:59.750099 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.929263 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:03 old-k8s-version-929157 kubelet[662]: E0305 04:39:03.749594 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.929470 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:11 old-k8s-version-929157 kubelet[662]: E0305 04:39:11.749896 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.929818 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:15 old-k8s-version-929157 kubelet[662]: E0305 04:39:15.749647 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.930021 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:24 old-k8s-version-929157 kubelet[662]: E0305 04:39:24.750162 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.930364 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:26 old-k8s-version-929157 kubelet[662]: E0305 04:39:26.753574 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.930572 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:35 old-k8s-version-929157 kubelet[662]: E0305 04:39:35.750670 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.930917 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:39 old-k8s-version-929157 kubelet[662]: E0305 04:39:39.749986 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.931121 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:46 old-k8s-version-929157 kubelet[662]: E0305 04:39:46.750507 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.931566 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:51 old-k8s-version-929157 kubelet[662]: E0305 04:39:51.749596 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.931781 2604473 logs.go:138] Found kubelet problem: Mar 05 04:39:58 old-k8s-version-929157 kubelet[662]: E0305 04:39:58.753750 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.932124 2604473 logs.go:138] Found kubelet problem: Mar 05 04:40:03 old-k8s-version-929157 kubelet[662]: E0305 04:40:03.749667 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:18.932326 2604473 logs.go:138] Found kubelet problem: Mar 05 04:40:09 old-k8s-version-929157 kubelet[662]: E0305 04:40:09.750261 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:18.932680 2604473 logs.go:138] Found kubelet problem: Mar 05 04:40:17 old-k8s-version-929157 kubelet[662]: E0305 04:40:17.764535 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
I0305 04:40:18.932710 2604473 logs.go:123] Gathering logs for etcd [3dd18badcf58a90e38fe4db8a253226bf84cf241094215382ff4b9cc24ee04a5] ...
I0305 04:40:18.932738 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 3dd18badcf58a90e38fe4db8a253226bf84cf241094215382ff4b9cc24ee04a5"
I0305 04:40:19.010200 2604473 logs.go:123] Gathering logs for coredns [508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99] ...
I0305 04:40:19.010362 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99"
I0305 04:40:19.064284 2604473 logs.go:123] Gathering logs for kube-controller-manager [9a1fcc970baf72fae5fc3eb76833e0232c8f51d7938d48ea284de0940c3a961c] ...
I0305 04:40:19.064350 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 9a1fcc970baf72fae5fc3eb76833e0232c8f51d7938d48ea284de0940c3a961c"
I0305 04:40:19.127144 2604473 logs.go:123] Gathering logs for kube-controller-manager [e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341] ...
I0305 04:40:19.127178 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341"
I0305 04:40:19.217413 2604473 logs.go:123] Gathering logs for kubernetes-dashboard [4c1bed4d5ba29d0be56313d0a71a5ec6110f707ff84f3f6e34f4028a3d82e073] ...
I0305 04:40:19.217494 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 4c1bed4d5ba29d0be56313d0a71a5ec6110f707ff84f3f6e34f4028a3d82e073"
I0305 04:40:19.293327 2604473 logs.go:123] Gathering logs for kube-proxy [acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f] ...
I0305 04:40:19.293409 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f"
I0305 04:40:19.358594 2604473 logs.go:123] Gathering logs for kindnet [feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232] ...
I0305 04:40:19.358673 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232"
I0305 04:40:19.411252 2604473 logs.go:123] Gathering logs for dmesg ...
I0305 04:40:19.411335 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0305 04:40:19.430380 2604473 logs.go:123] Gathering logs for kube-apiserver [fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259] ...
I0305 04:40:19.430513 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259"
I0305 04:40:19.504401 2604473 logs.go:123] Gathering logs for etcd [35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605] ...
I0305 04:40:19.504432 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605"
I0305 04:40:19.555953 2604473 logs.go:123] Gathering logs for coredns [e0dbacaa8e30885944ab46b688f9bbefac308f823788712e6c14ecdc1556f552] ...
I0305 04:40:19.556023 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 e0dbacaa8e30885944ab46b688f9bbefac308f823788712e6c14ecdc1556f552"
I0305 04:40:19.606777 2604473 logs.go:123] Gathering logs for kube-scheduler [a5d43d0d8a5d3f1e42ca4c1b9ec4dd6a812c6454f3858ec238850f1190bce93c] ...
I0305 04:40:19.606856 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 a5d43d0d8a5d3f1e42ca4c1b9ec4dd6a812c6454f3858ec238850f1190bce93c"
I0305 04:40:19.655430 2604473 logs.go:123] Gathering logs for kube-proxy [dd4135e2a8d47ee5af337a3314218c302ed97e8f2c878263df59d7b367ea4443] ...
I0305 04:40:19.655537 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 dd4135e2a8d47ee5af337a3314218c302ed97e8f2c878263df59d7b367ea4443"
I0305 04:40:19.712578 2604473 logs.go:123] Gathering logs for containerd ...
I0305 04:40:19.712646 2604473 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0305 04:40:19.792651 2604473 out.go:304] Setting ErrFile to fd 2...
I0305 04:40:19.792908 2604473 out.go:338] TERM=,COLORTERM=, which probably does not support color
W0305 04:40:19.792978 2604473 out.go:239] X Problems detected in kubelet:
W0305 04:40:19.793013 2604473 out.go:239] Mar 05 04:39:51 old-k8s-version-929157 kubelet[662]: E0305 04:39:51.749596 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:19.793047 2604473 out.go:239] Mar 05 04:39:58 old-k8s-version-929157 kubelet[662]: E0305 04:39:58.753750 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:19.793090 2604473 out.go:239] Mar 05 04:40:03 old-k8s-version-929157 kubelet[662]: E0305 04:40:03.749667 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
W0305 04:40:19.793123 2604473 out.go:239] Mar 05 04:40:09 old-k8s-version-929157 kubelet[662]: E0305 04:40:09.750261 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0305 04:40:19.793152 2604473 out.go:239] Mar 05 04:40:17 old-k8s-version-929157 kubelet[662]: E0305 04:40:17.764535 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
I0305 04:40:19.793189 2604473 out.go:304] Setting ErrFile to fd 2...
I0305 04:40:19.793210 2604473 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0305 04:40:18.426248 2614612 out.go:204] - Generating certificates and keys ...
I0305 04:40:18.426343 2614612 kubeadm.go:309] [certs] Using existing ca certificate authority
I0305 04:40:18.426413 2614612 kubeadm.go:309] [certs] Using existing apiserver certificate and key on disk
I0305 04:40:19.103670 2614612 kubeadm.go:309] [certs] Generating "apiserver-kubelet-client" certificate and key
I0305 04:40:19.926362 2614612 kubeadm.go:309] [certs] Generating "front-proxy-ca" certificate and key
I0305 04:40:20.279410 2614612 kubeadm.go:309] [certs] Generating "front-proxy-client" certificate and key
I0305 04:40:20.462010 2614612 kubeadm.go:309] [certs] Generating "etcd/ca" certificate and key
I0305 04:40:21.154626 2614612 kubeadm.go:309] [certs] Generating "etcd/server" certificate and key
I0305 04:40:21.154979 2614612 kubeadm.go:309] [certs] etcd/server serving cert is signed for DNS names [embed-certs-046985 localhost] and IPs [192.168.85.2 127.0.0.1 ::1]
I0305 04:40:21.442051 2614612 kubeadm.go:309] [certs] Generating "etcd/peer" certificate and key
I0305 04:40:21.444020 2614612 kubeadm.go:309] [certs] etcd/peer serving cert is signed for DNS names [embed-certs-046985 localhost] and IPs [192.168.85.2 127.0.0.1 ::1]
I0305 04:40:21.902846 2614612 kubeadm.go:309] [certs] Generating "etcd/healthcheck-client" certificate and key
I0305 04:40:22.352814 2614612 kubeadm.go:309] [certs] Generating "apiserver-etcd-client" certificate and key
I0305 04:40:23.266211 2614612 kubeadm.go:309] [certs] Generating "sa" key and public key
I0305 04:40:23.266298 2614612 kubeadm.go:309] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0305 04:40:24.322643 2614612 kubeadm.go:309] [kubeconfig] Writing "admin.conf" kubeconfig file
I0305 04:40:25.161931 2614612 kubeadm.go:309] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0305 04:40:25.380778 2614612 kubeadm.go:309] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0305 04:40:25.705150 2614612 kubeadm.go:309] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0305 04:40:25.705882 2614612 kubeadm.go:309] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0305 04:40:25.710471 2614612 kubeadm.go:309] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0305 04:40:25.713270 2614612 out.go:204] - Booting up control plane ...
I0305 04:40:25.713399 2614612 kubeadm.go:309] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0305 04:40:25.713495 2614612 kubeadm.go:309] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0305 04:40:25.713569 2614612 kubeadm.go:309] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0305 04:40:25.725261 2614612 kubeadm.go:309] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0305 04:40:25.726630 2614612 kubeadm.go:309] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0305 04:40:25.726691 2614612 kubeadm.go:309] [kubelet-start] Starting the kubelet
I0305 04:40:25.838908 2614612 kubeadm.go:309] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0305 04:40:29.793931 2604473 api_server.go:253] Checking apiserver healthz at https://192.168.76.2:8443/healthz ...
I0305 04:40:29.810892 2604473 api_server.go:279] https://192.168.76.2:8443/healthz returned 200:
ok
I0305 04:40:29.813726 2604473 out.go:177]
W0305 04:40:29.815636 2604473 out.go:239] X Exiting due to K8S_UNHEALTHY_CONTROL_PLANE: wait 6m0s for node: wait for healthy API server: controlPlane never updated to v1.20.0
W0305 04:40:29.815818 2604473 out.go:239] * Suggestion: Control Plane could not update, try minikube delete --all --purge
W0305 04:40:29.815873 2604473 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/11417
W0305 04:40:29.815906 2604473 out.go:239] *
W0305 04:40:29.817379 2604473 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0305 04:40:29.820417 2604473 out.go:177]
==> container status <==
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
046ee87d1e770 523cad1a4df73 2 minutes ago Exited dashboard-metrics-scraper 5 771a41a37052d dashboard-metrics-scraper-8d5bb5db8-4qrv5
9628a1e10f7ea ba04bb24b9575 4 minutes ago Running storage-provisioner 2 03366cc8520b7 storage-provisioner
4c1bed4d5ba29 20b332c9a70d8 5 minutes ago Running kubernetes-dashboard 0 dd80fd1c67cd3 kubernetes-dashboard-cd95d586-4zczx
6af9892bf6b5a 1611cd07b61d5 5 minutes ago Running busybox 1 d8b55aeaff1b6 busybox
cf35c688afe2b ba04bb24b9575 5 minutes ago Exited storage-provisioner 1 03366cc8520b7 storage-provisioner
dd4135e2a8d47 25a5233254979 5 minutes ago Running kube-proxy 1 48ee71cdfb727 kube-proxy-dxjrs
767fb3c810f4e 4740c1948d3fc 5 minutes ago Running kindnet-cni 1 bda2b08635919 kindnet-mmd8m
e0dbacaa8e308 db91994f4ee8f 5 minutes ago Running coredns 1 665d8334cf8b4 coredns-74ff55c5b-k5sj5
3dd18badcf58a 05b738aa1bc63 5 minutes ago Running etcd 1 a2a7665c6d4ea etcd-old-k8s-version-929157
9a1fcc970baf7 1df8a2b116bd1 5 minutes ago Running kube-controller-manager 1 a582158627dea kube-controller-manager-old-k8s-version-929157
a5d43d0d8a5d3 e7605f88f17d6 5 minutes ago Running kube-scheduler 1 10c91b4f77125 kube-scheduler-old-k8s-version-929157
71cb2e24b42a7 2c08bbbc02d3a 5 minutes ago Running kube-apiserver 1 bceb8a3cd3ca3 kube-apiserver-old-k8s-version-929157
c0e84e91bd6cc 1611cd07b61d5 6 minutes ago Exited busybox 0 2571c8e2439e7 busybox
508e3c5f81988 db91994f4ee8f 7 minutes ago Exited coredns 0 a1648983ececb coredns-74ff55c5b-k5sj5
acda2948fe9e0 25a5233254979 8 minutes ago Exited kube-proxy 0 fc771e17ef5af kube-proxy-dxjrs
feaa0468ae0f0 4740c1948d3fc 8 minutes ago Exited kindnet-cni 0 84b015ab7c42c kindnet-mmd8m
fc4de41b27b10 2c08bbbc02d3a 8 minutes ago Exited kube-apiserver 0 d264942afdb12 kube-apiserver-old-k8s-version-929157
27c1ecdfa4d5d e7605f88f17d6 8 minutes ago Exited kube-scheduler 0 8224787bb2963 kube-scheduler-old-k8s-version-929157
35a96501a5e8d 05b738aa1bc63 8 minutes ago Exited etcd 0 25ce0eef3dd33 etcd-old-k8s-version-929157
e34340acf6cb4 1df8a2b116bd1 8 minutes ago Exited kube-controller-manager 0 9ac2ca7606999 kube-controller-manager-old-k8s-version-929157
==> containerd <==
Mar 05 04:36:50 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:36:50.773031288Z" level=info msg="CreateContainer within sandbox \"771a41a37052d13c3f10f00c714c92256fb6503d64cceeab1898c4d287889095\" for &ContainerMetadata{Name:dashboard-metrics-scraper,Attempt:4,} returns container id \"2751ee5fddecebad70531151f68598f145d15bc4d64783338bd5e943c7d5f00e\""
Mar 05 04:36:50 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:36:50.775692386Z" level=info msg="StartContainer for \"2751ee5fddecebad70531151f68598f145d15bc4d64783338bd5e943c7d5f00e\""
Mar 05 04:36:50 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:36:50.841413537Z" level=info msg="StartContainer for \"2751ee5fddecebad70531151f68598f145d15bc4d64783338bd5e943c7d5f00e\" returns successfully"
Mar 05 04:36:50 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:36:50.871921277Z" level=info msg="shim disconnected" id=2751ee5fddecebad70531151f68598f145d15bc4d64783338bd5e943c7d5f00e
Mar 05 04:36:50 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:36:50.872224253Z" level=warning msg="cleaning up after shim disconnected" id=2751ee5fddecebad70531151f68598f145d15bc4d64783338bd5e943c7d5f00e namespace=k8s.io
Mar 05 04:36:50 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:36:50.872324829Z" level=info msg="cleaning up dead shim"
Mar 05 04:36:50 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:36:50.889282318Z" level=warning msg="cleanup warnings time=\"2024-03-05T04:36:50Z\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=3068 runtime=io.containerd.runc.v2\n"
Mar 05 04:36:51 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:36:51.313830009Z" level=info msg="RemoveContainer for \"a46ba9c80599a3b4418248943d38a854da205747fec80b8cb4f0c5378aa10a2c\""
Mar 05 04:36:51 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:36:51.320048463Z" level=info msg="RemoveContainer for \"a46ba9c80599a3b4418248943d38a854da205747fec80b8cb4f0c5378aa10a2c\" returns successfully"
Mar 05 04:37:39 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:37:39.751102810Z" level=info msg="PullImage \"fake.domain/registry.k8s.io/echoserver:1.4\""
Mar 05 04:37:39 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:37:39.755377129Z" level=info msg="trying next host" error="failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host" host=fake.domain
Mar 05 04:37:39 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:37:39.757103307Z" level=error msg="PullImage \"fake.domain/registry.k8s.io/echoserver:1.4\" failed" error="failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
Mar 05 04:38:17 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:38:17.751839399Z" level=info msg="CreateContainer within sandbox \"771a41a37052d13c3f10f00c714c92256fb6503d64cceeab1898c4d287889095\" for container &ContainerMetadata{Name:dashboard-metrics-scraper,Attempt:5,}"
Mar 05 04:38:17 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:38:17.768491063Z" level=info msg="CreateContainer within sandbox \"771a41a37052d13c3f10f00c714c92256fb6503d64cceeab1898c4d287889095\" for &ContainerMetadata{Name:dashboard-metrics-scraper,Attempt:5,} returns container id \"046ee87d1e770b9dc69cc14ba8528b04075275c2b3b43e18ba9b5059c6883118\""
Mar 05 04:38:17 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:38:17.769220005Z" level=info msg="StartContainer for \"046ee87d1e770b9dc69cc14ba8528b04075275c2b3b43e18ba9b5059c6883118\""
Mar 05 04:38:17 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:38:17.828712581Z" level=info msg="StartContainer for \"046ee87d1e770b9dc69cc14ba8528b04075275c2b3b43e18ba9b5059c6883118\" returns successfully"
Mar 05 04:38:17 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:38:17.852764469Z" level=info msg="shim disconnected" id=046ee87d1e770b9dc69cc14ba8528b04075275c2b3b43e18ba9b5059c6883118
Mar 05 04:38:17 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:38:17.852956637Z" level=warning msg="cleaning up after shim disconnected" id=046ee87d1e770b9dc69cc14ba8528b04075275c2b3b43e18ba9b5059c6883118 namespace=k8s.io
Mar 05 04:38:17 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:38:17.852982614Z" level=info msg="cleaning up dead shim"
Mar 05 04:38:17 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:38:17.861377441Z" level=warning msg="cleanup warnings time=\"2024-03-05T04:38:17Z\" level=info msg=\"starting signal loop\" namespace=k8s.io pid=3305 runtime=io.containerd.runc.v2\n"
Mar 05 04:38:18 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:38:18.525256539Z" level=info msg="RemoveContainer for \"2751ee5fddecebad70531151f68598f145d15bc4d64783338bd5e943c7d5f00e\""
Mar 05 04:38:18 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:38:18.531255887Z" level=info msg="RemoveContainer for \"2751ee5fddecebad70531151f68598f145d15bc4d64783338bd5e943c7d5f00e\" returns successfully"
Mar 05 04:40:23 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:40:23.751851595Z" level=info msg="PullImage \"fake.domain/registry.k8s.io/echoserver:1.4\""
Mar 05 04:40:23 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:40:23.757661030Z" level=info msg="trying next host" error="failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host" host=fake.domain
Mar 05 04:40:23 old-k8s-version-929157 containerd[568]: time="2024-03-05T04:40:23.759423966Z" level=error msg="PullImage \"fake.domain/registry.k8s.io/echoserver:1.4\" failed" error="failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
==> coredns [508e3c5f819885ebfbb0fc3f6732f44aa83e148cede00e6840e87aef720fac99] <==
.:53
[INFO] plugin/reload: Running configuration MD5 = b494d968e357ba1b925cee838fbd78ed
CoreDNS-1.7.0
linux/arm64, go1.14.4, f59c03d
[INFO] 127.0.0.1:41392 - 6540 "HINFO IN 2380660094136512821.2363802588503376728. udp 57 false 512" NXDOMAIN qr,rd,ra 57 0.012516169s
==> coredns [e0dbacaa8e30885944ab46b688f9bbefac308f823788712e6c14ecdc1556f552] <==
.:53
[INFO] plugin/reload: Running configuration MD5 = b494d968e357ba1b925cee838fbd78ed
CoreDNS-1.7.0
linux/arm64, go1.14.4, f59c03d
[INFO] 127.0.0.1:57723 - 55193 "HINFO IN 8120534617678173146.4130102372277805488. udp 57 false 512" NXDOMAIN qr,rd,ra 57 0.023168115s
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
I0305 04:35:20.827693 1 trace.go:116] Trace[939984059]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125 (started: 2024-03-05 04:34:50.824552903 +0000 UTC m=+0.051380447) (total time: 30.001076838s):
Trace[939984059]: [30.001076838s] [30.001076838s] END
I0305 04:35:20.827701 1 trace.go:116] Trace[2019727887]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125 (started: 2024-03-05 04:34:50.823337743 +0000 UTC m=+0.050165271) (total time: 30.002074903s):
Trace[2019727887]: [30.002074903s] [30.002074903s] END
E0305 04:35:20.829227 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Get "https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
E0305 04:35:20.829261 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
I0305 04:35:20.827759 1 trace.go:116] Trace[1427131847]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125 (started: 2024-03-05 04:34:50.823341009 +0000 UTC m=+0.050168529) (total time: 30.002122967s):
Trace[1427131847]: [30.002122967s] [30.002122967s] END
E0305 04:35:20.829413 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
==> describe nodes <==
Name: old-k8s-version-929157
Roles: control-plane,master
Labels: beta.kubernetes.io/arch=arm64
beta.kubernetes.io/os=linux
kubernetes.io/arch=arm64
kubernetes.io/hostname=old-k8s-version-929157
kubernetes.io/os=linux
minikube.k8s.io/commit=c10843c2362d401e82b228acf1f340a428845477
minikube.k8s.io/name=old-k8s-version-929157
minikube.k8s.io/primary=true
minikube.k8s.io/updated_at=2024_03_05T04_31_49_0700
minikube.k8s.io/version=v1.32.0
node-role.kubernetes.io/control-plane=
node-role.kubernetes.io/master=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Tue, 05 Mar 2024 04:31:45 +0000
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: old-k8s-version-929157
AcquireTime: <unset>
RenewTime: Tue, 05 Mar 2024 04:40:31 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Tue, 05 Mar 2024 04:35:42 +0000 Tue, 05 Mar 2024 04:31:39 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Tue, 05 Mar 2024 04:35:42 +0000 Tue, 05 Mar 2024 04:31:39 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Tue, 05 Mar 2024 04:35:42 +0000 Tue, 05 Mar 2024 04:31:39 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Tue, 05 Mar 2024 04:35:42 +0000 Tue, 05 Mar 2024 04:32:04 +0000 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.168.76.2
Hostname: old-k8s-version-929157
Capacity:
cpu: 2
ephemeral-storage: 203034800Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
hugepages-32Mi: 0
hugepages-64Ki: 0
memory: 8022500Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 203034800Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
hugepages-32Mi: 0
hugepages-64Ki: 0
memory: 8022500Ki
pods: 110
System Info:
Machine ID: 553111b1cab743fca26e846a44844beb
System UUID: 43faf8d1-a8f0-40ff-bf4f-aa241dc29c5d
Boot ID: 82232e04-3c34-4f6a-8fc1-2f07b9719df4
Kernel Version: 5.15.0-1055-aws
OS Image: Ubuntu 22.04.3 LTS
Operating System: linux
Architecture: arm64
Container Runtime Version: containerd://1.6.28
Kubelet Version: v1.20.0
Kube-Proxy Version: v1.20.0
PodCIDR: 10.244.0.0/24
PodCIDRs: 10.244.0.0/24
Non-terminated Pods: (12 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
default busybox 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 6m35s
kube-system coredns-74ff55c5b-k5sj5 100m (5%!)(MISSING) 0 (0%!)(MISSING) 70Mi (0%!)(MISSING) 170Mi (2%!)(MISSING) 8m28s
kube-system etcd-old-k8s-version-929157 100m (5%!)(MISSING) 0 (0%!)(MISSING) 100Mi (1%!)(MISSING) 0 (0%!)(MISSING) 8m35s
kube-system kindnet-mmd8m 100m (5%!)(MISSING) 100m (5%!)(MISSING) 50Mi (0%!)(MISSING) 50Mi (0%!)(MISSING) 8m28s
kube-system kube-apiserver-old-k8s-version-929157 250m (12%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 8m35s
kube-system kube-controller-manager-old-k8s-version-929157 200m (10%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 8m35s
kube-system kube-proxy-dxjrs 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 8m28s
kube-system kube-scheduler-old-k8s-version-929157 100m (5%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 8m35s
kube-system metrics-server-9975d5f86-wjtl9 100m (5%!)(MISSING) 0 (0%!)(MISSING) 200Mi (2%!)(MISSING) 0 (0%!)(MISSING) 6m25s
kube-system storage-provisioner 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 8m27s
kubernetes-dashboard dashboard-metrics-scraper-8d5bb5db8-4qrv5 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 5m26s
kubernetes-dashboard kubernetes-dashboard-cd95d586-4zczx 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 5m26s
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 950m (47%!)(MISSING) 100m (5%!)(MISSING)
memory 420Mi (5%!)(MISSING) 220Mi (2%!)(MISSING)
ephemeral-storage 100Mi (0%!)(MISSING) 0 (0%!)(MISSING)
hugepages-1Gi 0 (0%!)(MISSING) 0 (0%!)(MISSING)
hugepages-2Mi 0 (0%!)(MISSING) 0 (0%!)(MISSING)
hugepages-32Mi 0 (0%!)(MISSING) 0 (0%!)(MISSING)
hugepages-64Ki 0 (0%!)(MISSING) 0 (0%!)(MISSING)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeHasSufficientMemory 8m55s (x5 over 8m55s) kubelet Node old-k8s-version-929157 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 8m55s (x4 over 8m55s) kubelet Node old-k8s-version-929157 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 8m55s (x4 over 8m55s) kubelet Node old-k8s-version-929157 status is now: NodeHasSufficientPID
Normal Starting 8m36s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 8m35s kubelet Node old-k8s-version-929157 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 8m35s kubelet Node old-k8s-version-929157 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 8m35s kubelet Node old-k8s-version-929157 status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 8m35s kubelet Updated Node Allocatable limit across pods
Normal NodeReady 8m28s kubelet Node old-k8s-version-929157 status is now: NodeReady
Normal Starting 8m26s kube-proxy Starting kube-proxy.
Normal Starting 5m56s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 5m56s (x8 over 5m56s) kubelet Node old-k8s-version-929157 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 5m56s (x8 over 5m56s) kubelet Node old-k8s-version-929157 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 5m56s (x7 over 5m56s) kubelet Node old-k8s-version-929157 status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 5m56s kubelet Updated Node Allocatable limit across pods
Normal Starting 5m40s kube-proxy Starting kube-proxy.
==> dmesg <==
[ +0.001269] FS-Cache: O-key=[8] '6379ed0000000000'
[ +0.000735] FS-Cache: N-cookie c=0000008a [p=00000081 fl=2 nc=0 na=1]
[ +0.001060] FS-Cache: N-cookie d=000000001135e554{9p.inode} n=00000000598d7b07
[ +0.001142] FS-Cache: N-key=[8] '6379ed0000000000'
[ +0.006848] FS-Cache: Duplicate cookie detected
[ +0.000713] FS-Cache: O-cookie c=00000083 [p=00000081 fl=226 nc=0 na=1]
[ +0.001156] FS-Cache: O-cookie d=000000001135e554{9p.inode} n=000000004b0dbb2d
[ +0.001082] FS-Cache: O-key=[8] '6379ed0000000000'
[ +0.000807] FS-Cache: N-cookie c=0000008b [p=00000081 fl=2 nc=0 na=1]
[ +0.000970] FS-Cache: N-cookie d=000000001135e554{9p.inode} n=00000000e5445670
[ +0.001253] FS-Cache: N-key=[8] '6379ed0000000000'
[ +2.664525] FS-Cache: Duplicate cookie detected
[ +0.000701] FS-Cache: O-cookie c=00000082 [p=00000081 fl=226 nc=0 na=1]
[ +0.000985] FS-Cache: O-cookie d=000000001135e554{9p.inode} n=00000000af72d377
[ +0.001094] FS-Cache: O-key=[8] '6279ed0000000000'
[ +0.000715] FS-Cache: N-cookie c=0000008d [p=00000081 fl=2 nc=0 na=1]
[ +0.000952] FS-Cache: N-cookie d=000000001135e554{9p.inode} n=00000000598d7b07
[ +0.001047] FS-Cache: N-key=[8] '6279ed0000000000'
[ +0.329613] FS-Cache: Duplicate cookie detected
[ +0.000735] FS-Cache: O-cookie c=00000087 [p=00000081 fl=226 nc=0 na=1]
[ +0.001115] FS-Cache: O-cookie d=000000001135e554{9p.inode} n=000000005e4d73f0
[ +0.001059] FS-Cache: O-key=[8] '6879ed0000000000'
[ +0.000757] FS-Cache: N-cookie c=0000008e [p=00000081 fl=2 nc=0 na=1]
[ +0.000965] FS-Cache: N-cookie d=000000001135e554{9p.inode} n=00000000bced3eea
[ +0.001086] FS-Cache: N-key=[8] '6879ed0000000000'
==> etcd [35a96501a5e8d6b8fc7fcf556d7b7b56f9bc1631748996d36eec53b8a88eb605] <==
raft2024/03/05 04:31:38 INFO: ea7e25599daad906 became candidate at term 2
raft2024/03/05 04:31:38 INFO: ea7e25599daad906 received MsgVoteResp from ea7e25599daad906 at term 2
raft2024/03/05 04:31:38 INFO: ea7e25599daad906 became leader at term 2
raft2024/03/05 04:31:38 INFO: raft.node: ea7e25599daad906 elected leader ea7e25599daad906 at term 2
2024-03-05 04:31:38.546263 I | etcdserver: published {Name:old-k8s-version-929157 ClientURLs:[https://192.168.76.2:2379]} to cluster 6f20f2c4b2fb5f8a
2024-03-05 04:31:38.546397 I | etcdserver: setting up the initial cluster version to 3.4
2024-03-05 04:31:38.546453 I | embed: ready to serve client requests
2024-03-05 04:31:38.548607 I | embed: serving client requests on 192.168.76.2:2379
2024-03-05 04:31:38.548679 I | embed: ready to serve client requests
2024-03-05 04:31:38.553699 I | embed: serving client requests on 127.0.0.1:2379
2024-03-05 04:31:38.554523 N | etcdserver/membership: set the initial cluster version to 3.4
2024-03-05 04:31:38.554587 I | etcdserver/api: enabled capabilities for version 3.4
2024-03-05 04:32:03.302385 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:32:08.008742 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:32:18.009869 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:32:28.008283 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:32:38.013205 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:32:48.010840 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:32:58.010343 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:33:08.008236 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:33:18.009044 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:33:28.009276 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:33:38.008217 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:33:48.008687 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:33:58.012175 I | etcdserver/api/etcdhttp: /health OK (status code 200)
==> etcd [3dd18badcf58a90e38fe4db8a253226bf84cf241094215382ff4b9cc24ee04a5] <==
2024-03-05 04:36:29.264575 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:36:39.264509 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:36:49.264458 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:36:59.264540 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:37:09.264557 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:37:19.264520 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:37:29.264519 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:37:39.264474 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:37:49.264484 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:37:59.264587 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:38:09.264543 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:38:19.264454 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:38:29.264436 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:38:39.264579 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:38:49.264540 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:38:59.264756 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:39:09.264776 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:39:19.264601 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:39:29.264593 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:39:39.276101 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:39:49.264684 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:39:59.264629 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:40:09.264636 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:40:19.264436 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-03-05 04:40:29.264636 I | etcdserver/api/etcdhttp: /health OK (status code 200)
==> kernel <==
04:40:32 up 10:22, 0 users, load average: 1.81, 2.14, 2.65
Linux old-k8s-version-929157 5.15.0-1055-aws #60~20.04.1-Ubuntu SMP Thu Feb 22 15:54:21 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.3 LTS"
==> kindnet [767fb3c810f4e121fae9120786476ac6029d586fd1a6203f8cf8917dbbaf13cb] <==
I0305 04:38:31.991967 1 main.go:227] handling current node
I0305 04:38:42.004392 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:38:42.004429 1 main.go:227] handling current node
I0305 04:38:52.011961 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:38:52.011995 1 main.go:227] handling current node
I0305 04:39:02.026303 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:39:02.026330 1 main.go:227] handling current node
I0305 04:39:12.032770 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:39:12.032802 1 main.go:227] handling current node
I0305 04:39:22.040090 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:39:22.040118 1 main.go:227] handling current node
I0305 04:39:32.047577 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:39:32.047607 1 main.go:227] handling current node
I0305 04:39:42.061088 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:39:42.061118 1 main.go:227] handling current node
I0305 04:39:52.066877 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:39:52.066907 1 main.go:227] handling current node
I0305 04:40:02.081661 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:40:02.081696 1 main.go:227] handling current node
I0305 04:40:12.085630 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:40:12.085663 1 main.go:227] handling current node
I0305 04:40:22.097163 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:40:22.097196 1 main.go:227] handling current node
I0305 04:40:32.111692 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:40:32.111732 1 main.go:227] handling current node
==> kindnet [feaa0468ae0f05495059f8d30d3dcd103e32bebd3dc76f5764eddce1349d9232] <==
podIP = 192.168.76.2
I0305 04:32:05.203341 1 main.go:116] setting mtu 1500 for CNI
I0305 04:32:05.203351 1 main.go:146] kindnetd IP family: "ipv4"
I0305 04:32:05.203361 1 main.go:150] noMask IPv4 subnets: [10.244.0.0/16]
I0305 04:32:35.431914 1 main.go:191] Failed to get nodes, retrying after error: Get "https://10.96.0.1:443/api/v1/nodes": dial tcp 10.96.0.1:443: i/o timeout
I0305 04:32:35.445533 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:32:35.445571 1 main.go:227] handling current node
I0305 04:32:45.502908 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:32:45.502938 1 main.go:227] handling current node
I0305 04:32:55.520339 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:32:55.520366 1 main.go:227] handling current node
I0305 04:33:05.529610 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:33:05.529637 1 main.go:227] handling current node
I0305 04:33:15.537642 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:33:15.537674 1 main.go:227] handling current node
I0305 04:33:25.553031 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:33:25.553064 1 main.go:227] handling current node
I0305 04:33:35.576147 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:33:35.576175 1 main.go:227] handling current node
I0305 04:33:45.588834 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:33:45.588865 1 main.go:227] handling current node
I0305 04:33:55.604201 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:33:55.604296 1 main.go:227] handling current node
I0305 04:34:05.622658 1 main.go:223] Handling node with IPs: map[192.168.76.2:{}]
I0305 04:34:05.623078 1 main.go:227] handling current node
==> kube-apiserver [71cb2e24b42a7f8d797580bf0b66535d6aa909eae6127ef3f669c69251df7f53] <==
I0305 04:37:16.471606 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0305 04:37:16.471625 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
W0305 04:37:51.092381 1 handler_proxy.go:102] no RequestInfo found in the context
E0305 04:37:51.092455 1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0305 04:37:51.092470 1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0305 04:37:55.682295 1 client.go:360] parsed scheme: "passthrough"
I0305 04:37:55.682347 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0305 04:37:55.682357 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0305 04:38:39.978311 1 client.go:360] parsed scheme: "passthrough"
I0305 04:38:39.978354 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0305 04:38:39.978362 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0305 04:39:16.615623 1 client.go:360] parsed scheme: "passthrough"
I0305 04:39:16.615669 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0305 04:39:16.615678 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
W0305 04:39:49.659270 1 handler_proxy.go:102] no RequestInfo found in the context
E0305 04:39:49.659354 1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0305 04:39:49.659363 1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0305 04:39:52.684203 1 client.go:360] parsed scheme: "passthrough"
I0305 04:39:52.684247 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0305 04:39:52.684256 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0305 04:40:22.819121 1 client.go:360] parsed scheme: "passthrough"
I0305 04:40:22.819178 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0305 04:40:22.819193 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
==> kube-apiserver [fc4de41b27b104cd009210204c51ada139d4c9fe1e36c58fb938607e9d2d1259] <==
I0305 04:31:46.370144 1 storage_scheduling.go:132] created PriorityClass system-cluster-critical with value 2000000000
I0305 04:31:46.370169 1 storage_scheduling.go:148] all system priority classes are created successfully or already exist.
I0305 04:31:46.845403 1 controller.go:606] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0305 04:31:46.891085 1 controller.go:606] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
W0305 04:31:47.032362 1 lease.go:233] Resetting endpoints for master service "kubernetes" to [192.168.76.2]
I0305 04:31:47.033382 1 controller.go:606] quota admission added evaluator for: endpoints
I0305 04:31:47.037399 1 controller.go:606] quota admission added evaluator for: endpointslices.discovery.k8s.io
I0305 04:31:47.463852 1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io
I0305 04:31:48.032577 1 controller.go:606] quota admission added evaluator for: serviceaccounts
I0305 04:31:48.512762 1 controller.go:606] quota admission added evaluator for: deployments.apps
I0305 04:31:48.576926 1 controller.go:606] quota admission added evaluator for: daemonsets.apps
I0305 04:32:04.077558 1 controller.go:606] quota admission added evaluator for: controllerrevisions.apps
I0305 04:32:04.103727 1 controller.go:606] quota admission added evaluator for: replicasets.apps
I0305 04:32:24.166103 1 client.go:360] parsed scheme: "passthrough"
I0305 04:32:24.166150 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0305 04:32:24.166158 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0305 04:32:56.458279 1 client.go:360] parsed scheme: "passthrough"
I0305 04:32:56.458405 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0305 04:32:56.458434 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0305 04:33:32.010312 1 client.go:360] parsed scheme: "passthrough"
I0305 04:33:32.010426 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0305 04:33:32.010455 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0305 04:34:07.310009 1 client.go:360] parsed scheme: "passthrough"
I0305 04:34:07.310048 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0305 04:34:07.310057 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
==> kube-controller-manager [9a1fcc970baf72fae5fc3eb76833e0232c8f51d7938d48ea284de0940c3a961c] <==
W0305 04:36:11.931527 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0305 04:36:38.816298 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0305 04:36:43.581969 1 request.go:655] Throttling request took 1.048430456s, request: GET:https://192.168.76.2:8443/apis/networking.k8s.io/v1?timeout=32s
W0305 04:36:44.433444 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0305 04:37:09.318140 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0305 04:37:16.083876 1 request.go:655] Throttling request took 1.043518368s, request: GET:https://192.168.76.2:8443/apis/extensions/v1beta1?timeout=32s
W0305 04:37:16.935363 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0305 04:37:39.820123 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0305 04:37:48.585754 1 request.go:655] Throttling request took 1.048265657s, request: GET:https://192.168.76.2:8443/apis/networking.k8s.io/v1beta1?timeout=32s
W0305 04:37:49.437277 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0305 04:38:10.321950 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0305 04:38:21.087746 1 request.go:655] Throttling request took 1.048318203s, request: GET:https://192.168.76.2:8443/apis/autoscaling/v1?timeout=32s
W0305 04:38:21.938986 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0305 04:38:40.823705 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0305 04:38:53.589537 1 request.go:655] Throttling request took 1.048169413s, request: GET:https://192.168.76.2:8443/apis/admissionregistration.k8s.io/v1beta1?timeout=32s
W0305 04:38:54.441150 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0305 04:39:11.325599 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0305 04:39:26.092225 1 request.go:655] Throttling request took 1.048586014s, request: GET:https://192.168.76.2:8443/apis/scheduling.k8s.io/v1?timeout=32s
W0305 04:39:26.943356 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0305 04:39:41.827334 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0305 04:39:58.595566 1 request.go:655] Throttling request took 1.049906032s, request: GET:https://192.168.76.2:8443/apis/apiextensions.k8s.io/v1beta1?timeout=32s
W0305 04:39:59.445775 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0305 04:40:12.329558 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0305 04:40:31.096154 1 request.go:655] Throttling request took 1.046291123s, request: GET:https://192.168.76.2:8443/apis/extensions/v1beta1?timeout=32s
W0305 04:40:31.948227 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
==> kube-controller-manager [e34340acf6cb490675bc9402004653346f60d5bbd2711282470cb2d5793b9341] <==
I0305 04:32:04.161445 1 event.go:291] "Event occurred" object="kube-system/coredns" kind="Deployment" apiVersion="apps/v1" type="Normal" reason="ScalingReplicaSet" message="Scaled up replica set coredns-74ff55c5b to 2"
I0305 04:32:04.175061 1 shared_informer.go:247] Caches are synced for attach detach
I0305 04:32:04.199236 1 shared_informer.go:247] Caches are synced for expand
I0305 04:32:04.212598 1 shared_informer.go:247] Caches are synced for resource quota
I0305 04:32:04.213865 1 event.go:291] "Event occurred" object="kube-system/kindnet" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: kindnet-mmd8m"
I0305 04:32:04.215782 1 event.go:291] "Event occurred" object="kube-system/kube-proxy" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: kube-proxy-dxjrs"
I0305 04:32:04.225935 1 shared_informer.go:247] Caches are synced for ClusterRoleAggregator
I0305 04:32:04.248306 1 event.go:291] "Event occurred" object="kube-system/coredns-74ff55c5b" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: coredns-74ff55c5b-2q77c"
I0305 04:32:04.298960 1 event.go:291] "Event occurred" object="kube-system/coredns-74ff55c5b" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: coredns-74ff55c5b-k5sj5"
E0305 04:32:04.327393 1 daemon_controller.go:320] kube-system/kube-proxy failed with : error storing status for daemon set &v1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-proxy", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"902f3a0a-eff6-4286-98b6-90e8be16dcb4", ResourceVersion:"259", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63845209908, loc:(*time.Location)(0x632eb80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string{"deprecated.daemonset.template.generation":"1"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kubeadm", Operation:"Update", APIVersion:"apps/v1", Time:(*v1.Time)(0x4001aee3e0), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0x4001aee400)}}}, Spec:v1.DaemonSetSpec{Selector:(*v1.
LabelSelector)(0x4001aee420), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"k8s-app":"kube-proxy"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"kube-proxy", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(nil), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.Gl
usterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(0x4001acd980), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil), Ephemeral:(*v1.EphemeralVolumeSource)(nil)}}, v1.Volume{Name:"xtables-lock", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0x4001aee
440), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeS
ource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil), Ephemeral:(*v1.EphemeralVolumeSource)(nil)}}, v1.Volume{Name:"lib-modules", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0x4001aee460), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil),
AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil), Ephemeral:(*v1.EphemeralVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"kube-proxy", Image:"k8s.gcr.io/kube-proxy:v1.20.0", Command:[]string{"/usr/local/bin/kube-proxy", "--config=/var/lib/kube-proxy/config.conf", "--hostname-override=$(NODE_NAME)"}, Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"NODE_NAME", Value:"", ValueFrom:(*v1.EnvVarSource)(0x4001aee4a0)}}, Resources:v1.R
esourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"kube-proxy", ReadOnly:false, MountPath:"/var/lib/kube-proxy", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}, v1.VolumeMount{Name:"xtables-lock", ReadOnly:false, MountPath:"/run/xtables.lock", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}, v1.VolumeMount{Name:"lib-modules", ReadOnly:true, MountPath:"/lib/modules", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), StartupProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0x4001ab5500), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]v1.EphemeralContainer(nil), RestartPo
licy:"Always", TerminationGracePeriodSeconds:(*int64)(0x4001af04f8), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string{"kubernetes.io/os":"linux"}, ServiceAccountName:"kube-proxy", DeprecatedServiceAccount:"kube-proxy", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:true, HostPID:false, HostIPC:false, ShareProcessNamespace:(*bool)(nil), SecurityContext:(*v1.PodSecurityContext)(0x40003630a0), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"CriticalAddonsOnly", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}, v1.Toleration{Key:"", Operator:"Exists", Value:"", Effect:"", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"system-node-critical", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil), ReadinessGates:[]v1.PodReadinessGate(nil), Runtime
ClassName:(*string)(nil), EnableServiceLinks:(*bool)(nil), PreemptionPolicy:(*v1.PreemptionPolicy)(nil), Overhead:v1.ResourceList(nil), TopologySpreadConstraints:[]v1.TopologySpreadConstraint(nil), SetHostnameAsFQDN:(*bool)(nil)}}, UpdateStrategy:v1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1.RollingUpdateDaemonSet)(0x400011a788)}, MinReadySeconds:0, RevisionHistoryLimit:(*int32)(0x4001af0548)}, Status:v1.DaemonSetStatus{CurrentNumberScheduled:0, NumberMisscheduled:0, DesiredNumberScheduled:0, NumberReady:0, ObservedGeneration:0, UpdatedNumberScheduled:0, NumberAvailable:0, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.apps "kube-proxy": the object has been modified; please apply your changes to the latest version and try again
I0305 04:32:04.368478 1 shared_informer.go:240] Waiting for caches to sync for garbage collector
I0305 04:32:04.660971 1 shared_informer.go:247] Caches are synced for garbage collector
I0305 04:32:04.660998 1 garbagecollector.go:151] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
I0305 04:32:04.668882 1 shared_informer.go:247] Caches are synced for garbage collector
I0305 04:32:04.756206 1 request.go:655] Throttling request took 1.048416449s, request: GET:https://192.168.76.2:8443/apis/networking.k8s.io/v1beta1?timeout=32s
I0305 04:32:05.569503 1 shared_informer.go:240] Waiting for caches to sync for resource quota
I0305 04:32:05.569795 1 shared_informer.go:247] Caches are synced for resource quota
I0305 04:32:05.605971 1 event.go:291] "Event occurred" object="kube-system/coredns" kind="Deployment" apiVersion="apps/v1" type="Normal" reason="ScalingReplicaSet" message="Scaled down replica set coredns-74ff55c5b to 1"
I0305 04:32:05.624126 1 event.go:291] "Event occurred" object="kube-system/coredns-74ff55c5b" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulDelete" message="Deleted pod: coredns-74ff55c5b-2q77c"
I0305 04:32:09.076439 1 node_lifecycle_controller.go:1222] Controller detected that some Nodes are Ready. Exiting master disruption mode.
I0305 04:34:06.686348 1 event.go:291] "Event occurred" object="kube-system/metrics-server" kind="Deployment" apiVersion="apps/v1" type="Normal" reason="ScalingReplicaSet" message="Scaled up replica set metrics-server-9975d5f86 to 1"
I0305 04:34:06.762505 1 event.go:291] "Event occurred" object="kube-system/metrics-server-9975d5f86" kind="ReplicaSet" apiVersion="apps/v1" type="Warning" reason="FailedCreate" message="Error creating: pods \"metrics-server-9975d5f86-\" is forbidden: error looking up service account kube-system/metrics-server: serviceaccount \"metrics-server\" not found"
E0305 04:34:06.824825 1 replica_set.go:532] sync "kube-system/metrics-server-9975d5f86" failed with pods "metrics-server-9975d5f86-" is forbidden: error looking up service account kube-system/metrics-server: serviceaccount "metrics-server" not found
E0305 04:34:07.056705 1 clusterroleaggregation_controller.go:181] admin failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "admin": the object has been modified; please apply your changes to the latest version and try again
E0305 04:34:07.433785 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server could not find the requested resource
==> kube-proxy [acda2948fe9e022ad7085474bd9b447d0bf6993f1036c2959d97aa550c25b23f] <==
I0305 04:32:06.280884 1 node.go:172] Successfully retrieved node IP: 192.168.76.2
I0305 04:32:06.280979 1 server_others.go:142] kube-proxy node IP is an IPv4 address (192.168.76.2), assume IPv4 operation
W0305 04:32:06.302098 1 server_others.go:578] Unknown proxy mode "", assuming iptables proxy
I0305 04:32:06.302469 1 server_others.go:185] Using iptables Proxier.
I0305 04:32:06.304283 1 server.go:650] Version: v1.20.0
I0305 04:32:06.304891 1 config.go:315] Starting service config controller
I0305 04:32:06.304905 1 shared_informer.go:240] Waiting for caches to sync for service config
I0305 04:32:06.305901 1 config.go:224] Starting endpoint slice config controller
I0305 04:32:06.306068 1 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
I0305 04:32:06.405039 1 shared_informer.go:247] Caches are synced for service config
I0305 04:32:06.406319 1 shared_informer.go:247] Caches are synced for endpoint slice config
==> kube-proxy [dd4135e2a8d47ee5af337a3314218c302ed97e8f2c878263df59d7b367ea4443] <==
I0305 04:34:52.789213 1 node.go:172] Successfully retrieved node IP: 192.168.76.2
I0305 04:34:52.789276 1 server_others.go:142] kube-proxy node IP is an IPv4 address (192.168.76.2), assume IPv4 operation
W0305 04:34:52.821307 1 server_others.go:578] Unknown proxy mode "", assuming iptables proxy
I0305 04:34:52.821615 1 server_others.go:185] Using iptables Proxier.
I0305 04:34:52.822027 1 server.go:650] Version: v1.20.0
I0305 04:34:52.823436 1 config.go:224] Starting endpoint slice config controller
I0305 04:34:52.823622 1 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
I0305 04:34:52.823750 1 config.go:315] Starting service config controller
I0305 04:34:52.823828 1 shared_informer.go:240] Waiting for caches to sync for service config
I0305 04:34:52.923870 1 shared_informer.go:247] Caches are synced for endpoint slice config
I0305 04:34:52.923960 1 shared_informer.go:247] Caches are synced for service config
==> kube-scheduler [27c1ecdfa4d5d3514f657cefc09eda6d49f940ecd3fade119f69a18a84094636] <==
W0305 04:31:45.536833 1 requestheader_controller.go:193] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0305 04:31:45.536871 1 authentication.go:332] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0305 04:31:45.536891 1 authentication.go:333] Continuing without authentication configuration. This may treat all requests as anonymous.
W0305 04:31:45.536896 1 authentication.go:334] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0305 04:31:45.621943 1 secure_serving.go:197] Serving securely on 127.0.0.1:10259
I0305 04:31:45.622665 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0305 04:31:45.622687 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0305 04:31:45.622709 1 tlsconfig.go:240] Starting DynamicServingCertificateController
E0305 04:31:45.644684 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0305 04:31:45.645239 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0305 04:31:45.647755 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0305 04:31:45.647968 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0305 04:31:45.648263 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0305 04:31:45.648476 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0305 04:31:45.648823 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0305 04:31:45.649108 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0305 04:31:45.649470 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0305 04:31:45.649619 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0305 04:31:45.649714 1 reflector.go:138] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
E0305 04:31:45.651684 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0305 04:31:46.465157 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0305 04:31:46.507367 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0305 04:31:46.546466 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0305 04:31:46.655741 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
I0305 04:31:47.022788 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
==> kube-scheduler [a5d43d0d8a5d3f1e42ca4c1b9ec4dd6a812c6454f3858ec238850f1190bce93c] <==
I0305 04:34:40.107082 1 serving.go:331] Generated self-signed cert in-memory
W0305 04:34:48.568132 1 requestheader_controller.go:193] Unable to get configmap/extension-apiserver-authentication in kube-system. Usually fixed by 'kubectl create rolebinding -n kube-system ROLEBINDING_NAME --role=extension-apiserver-authentication-reader --serviceaccount=YOUR_NS:YOUR_SA'
W0305 04:34:48.568359 1 authentication.go:332] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0305 04:34:48.568455 1 authentication.go:333] Continuing without authentication configuration. This may treat all requests as anonymous.
W0305 04:34:48.568540 1 authentication.go:334] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0305 04:34:48.718936 1 secure_serving.go:197] Serving securely on 127.0.0.1:10259
I0305 04:34:48.719377 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0305 04:34:48.719539 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0305 04:34:48.719680 1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0305 04:34:48.920351 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
==> kubelet <==
Mar 05 04:39:03 old-k8s-version-929157 kubelet[662]: E0305 04:39:03.749594 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
Mar 05 04:39:11 old-k8s-version-929157 kubelet[662]: E0305 04:39:11.749896 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Mar 05 04:39:15 old-k8s-version-929157 kubelet[662]: I0305 04:39:15.749267 662 scope.go:95] [topologymanager] RemoveContainer - Container ID: 046ee87d1e770b9dc69cc14ba8528b04075275c2b3b43e18ba9b5059c6883118
Mar 05 04:39:15 old-k8s-version-929157 kubelet[662]: E0305 04:39:15.749647 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
Mar 05 04:39:24 old-k8s-version-929157 kubelet[662]: E0305 04:39:24.750162 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Mar 05 04:39:26 old-k8s-version-929157 kubelet[662]: I0305 04:39:26.751787 662 scope.go:95] [topologymanager] RemoveContainer - Container ID: 046ee87d1e770b9dc69cc14ba8528b04075275c2b3b43e18ba9b5059c6883118
Mar 05 04:39:26 old-k8s-version-929157 kubelet[662]: E0305 04:39:26.753574 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
Mar 05 04:39:35 old-k8s-version-929157 kubelet[662]: E0305 04:39:35.750670 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Mar 05 04:39:39 old-k8s-version-929157 kubelet[662]: I0305 04:39:39.749231 662 scope.go:95] [topologymanager] RemoveContainer - Container ID: 046ee87d1e770b9dc69cc14ba8528b04075275c2b3b43e18ba9b5059c6883118
Mar 05 04:39:39 old-k8s-version-929157 kubelet[662]: E0305 04:39:39.749986 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
Mar 05 04:39:46 old-k8s-version-929157 kubelet[662]: E0305 04:39:46.750507 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Mar 05 04:39:51 old-k8s-version-929157 kubelet[662]: I0305 04:39:51.749260 662 scope.go:95] [topologymanager] RemoveContainer - Container ID: 046ee87d1e770b9dc69cc14ba8528b04075275c2b3b43e18ba9b5059c6883118
Mar 05 04:39:51 old-k8s-version-929157 kubelet[662]: E0305 04:39:51.749596 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
Mar 05 04:39:58 old-k8s-version-929157 kubelet[662]: E0305 04:39:58.753750 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Mar 05 04:40:03 old-k8s-version-929157 kubelet[662]: I0305 04:40:03.749321 662 scope.go:95] [topologymanager] RemoveContainer - Container ID: 046ee87d1e770b9dc69cc14ba8528b04075275c2b3b43e18ba9b5059c6883118
Mar 05 04:40:03 old-k8s-version-929157 kubelet[662]: E0305 04:40:03.749667 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
Mar 05 04:40:09 old-k8s-version-929157 kubelet[662]: E0305 04:40:09.750261 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Mar 05 04:40:17 old-k8s-version-929157 kubelet[662]: I0305 04:40:17.764152 662 scope.go:95] [topologymanager] RemoveContainer - Container ID: 046ee87d1e770b9dc69cc14ba8528b04075275c2b3b43e18ba9b5059c6883118
Mar 05 04:40:17 old-k8s-version-929157 kubelet[662]: E0305 04:40:17.764535 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
Mar 05 04:40:23 old-k8s-version-929157 kubelet[662]: E0305 04:40:23.759779 662 remote_image.go:113] PullImage "fake.domain/registry.k8s.io/echoserver:1.4" from image service failed: rpc error: code = Unknown desc = failed to pull and unpack image "fake.domain/registry.k8s.io/echoserver:1.4": failed to resolve reference "fake.domain/registry.k8s.io/echoserver:1.4": failed to do request: Head "https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host
Mar 05 04:40:23 old-k8s-version-929157 kubelet[662]: E0305 04:40:23.759855 662 kuberuntime_image.go:51] Pull image "fake.domain/registry.k8s.io/echoserver:1.4" failed: rpc error: code = Unknown desc = failed to pull and unpack image "fake.domain/registry.k8s.io/echoserver:1.4": failed to resolve reference "fake.domain/registry.k8s.io/echoserver:1.4": failed to do request: Head "https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host
Mar 05 04:40:23 old-k8s-version-929157 kubelet[662]: E0305 04:40:23.760128 662 kuberuntime_manager.go:829] container &Container{Name:metrics-server,Image:fake.domain/registry.k8s.io/echoserver:1.4,Command:[],Args:[--cert-dir=/tmp --secure-port=4443 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=60s --kubelet-insecure-tls],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:https,HostPort:0,ContainerPort:4443,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{100 -3} {<nil>} 100m DecimalSI},memory: {{209715200 0} {<nil>} BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:tmp-dir,ReadOnly:false,MountPath:/tmp,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:metrics-server-token-z55cz,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:&Probe{Handler:Handler{Exec
:nil,HTTPGet:&HTTPGetAction{Path:/livez,Port:{1 0 https},Host:,Scheme:HTTPS,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,},ReadinessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/readyz,Port:{1 0 https},Host:,Scheme:HTTPS,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,},Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:nil,SELinuxOptions:nil,RunAsUser:*1000,RunAsNonRoot:*true,ReadOnlyRootFilesystem:*true,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod metrics-server-9975d5f86-wjtl9_kube-system(242a840
6-2a42-4a0c-ad66-997da922bf7d): ErrImagePull: rpc error: code = Unknown desc = failed to pull and unpack image "fake.domain/registry.k8s.io/echoserver:1.4": failed to resolve reference "fake.domain/registry.k8s.io/echoserver:1.4": failed to do request: Head "https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host
Mar 05 04:40:23 old-k8s-version-929157 kubelet[662]: E0305 04:40:23.760188 662 pod_workers.go:191] Error syncing pod 242a8406-2a42-4a0c-ad66-997da922bf7d ("metrics-server-9975d5f86-wjtl9_kube-system(242a8406-2a42-4a0c-ad66-997da922bf7d)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.76.1:53: no such host"
Mar 05 04:40:29 old-k8s-version-929157 kubelet[662]: I0305 04:40:29.749396 662 scope.go:95] [topologymanager] RemoveContainer - Container ID: 046ee87d1e770b9dc69cc14ba8528b04075275c2b3b43e18ba9b5059c6883118
Mar 05 04:40:29 old-k8s-version-929157 kubelet[662]: E0305 04:40:29.749732 662 pod_workers.go:191] Error syncing pod ada71ab7-57d5-454a-bbfb-edad70fc9746 ("dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-4qrv5_kubernetes-dashboard(ada71ab7-57d5-454a-bbfb-edad70fc9746)"
==> kubernetes-dashboard [4c1bed4d5ba29d0be56313d0a71a5ec6110f707ff84f3f6e34f4028a3d82e073] <==
2024/03/05 04:35:15 Using namespace: kubernetes-dashboard
2024/03/05 04:35:15 Using in-cluster config to connect to apiserver
2024/03/05 04:35:15 Using secret token for csrf signing
2024/03/05 04:35:15 Initializing csrf token from kubernetes-dashboard-csrf secret
2024/03/05 04:35:15 Empty token. Generating and storing in a secret kubernetes-dashboard-csrf
2024/03/05 04:35:15 Successful initial request to the apiserver, version: v1.20.0
2024/03/05 04:35:15 Generating JWE encryption key
2024/03/05 04:35:15 New synchronizer has been registered: kubernetes-dashboard-key-holder-kubernetes-dashboard. Starting
2024/03/05 04:35:15 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kubernetes-dashboard
2024/03/05 04:35:15 Initializing JWE encryption key from synchronized object
2024/03/05 04:35:15 Creating in-cluster Sidecar client
2024/03/05 04:35:15 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/03/05 04:35:15 Serving insecurely on HTTP port: 9090
2024/03/05 04:35:45 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/03/05 04:36:15 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/03/05 04:36:45 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/03/05 04:37:15 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/03/05 04:37:45 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/03/05 04:38:15 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/03/05 04:38:45 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/03/05 04:39:15 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/03/05 04:39:45 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/03/05 04:40:15 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/03/05 04:35:15 Starting overwatch
==> storage-provisioner [9628a1e10f7ea4fab2791b498ecbe6f8059145a340d5e9e09c8e2c4ea87c21ff] <==
I0305 04:35:37.117660 1 storage_provisioner.go:116] Initializing the minikube storage provisioner...
I0305 04:35:37.162361 1 storage_provisioner.go:141] Storage provisioner initialized, now starting service!
I0305 04:35:37.162408 1 leaderelection.go:243] attempting to acquire leader lease kube-system/k8s.io-minikube-hostpath...
I0305 04:35:54.671353 1 leaderelection.go:253] successfully acquired lease kube-system/k8s.io-minikube-hostpath
I0305 04:35:54.671990 1 event.go:282] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"k8s.io-minikube-hostpath", UID:"0f9d924d-6931-4bab-bbe7-dbd813ead74e", APIVersion:"v1", ResourceVersion:"883", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' old-k8s-version-929157_00816114-a99b-4cb6-bb9f-d463bc6a80f4 became leader
I0305 04:35:54.672163 1 controller.go:835] Starting provisioner controller k8s.io/minikube-hostpath_old-k8s-version-929157_00816114-a99b-4cb6-bb9f-d463bc6a80f4!
I0305 04:35:54.773388 1 controller.go:884] Started provisioner controller k8s.io/minikube-hostpath_old-k8s-version-929157_00816114-a99b-4cb6-bb9f-d463bc6a80f4!
==> storage-provisioner [cf35c688afe2b4af1fb0bb251e7c94c0f42f0ef8989f9ed8720915397d2571d7] <==
I0305 04:34:52.669682 1 storage_provisioner.go:116] Initializing the minikube storage provisioner...
F0305 04:35:22.672006 1 main.go:39] error getting server version: Get "https://10.96.0.1:443/version?timeout=32s": dial tcp 10.96.0.1:443: i/o timeout
-- /stdout --
helpers_test.go:254: (dbg) Run: out/minikube-linux-arm64 status --format={{.APIServer}} -p old-k8s-version-929157 -n old-k8s-version-929157
helpers_test.go:261: (dbg) Run: kubectl --context old-k8s-version-929157 get po -o=jsonpath={.items[*].metadata.name} -A --field-selector=status.phase!=Running
helpers_test.go:272: non-running pods: metrics-server-9975d5f86-wjtl9
helpers_test.go:274: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/SecondStart]: describe non-running pods <======
helpers_test.go:277: (dbg) Run: kubectl --context old-k8s-version-929157 describe pod metrics-server-9975d5f86-wjtl9
helpers_test.go:277: (dbg) Non-zero exit: kubectl --context old-k8s-version-929157 describe pod metrics-server-9975d5f86-wjtl9: exit status 1 (122.014107ms)
** stderr **
Error from server (NotFound): pods "metrics-server-9975d5f86-wjtl9" not found
** /stderr **
helpers_test.go:279: kubectl --context old-k8s-version-929157 describe pod metrics-server-9975d5f86-wjtl9: exit status 1
--- FAIL: TestStartStop/group/old-k8s-version/serial/SecondStart (374.35s)