=== RUN TestStartStop/group/old-k8s-version/serial/SecondStart
start_stop_delete_test.go:256: (dbg) Run: out/minikube-linux-arm64 start -p old-k8s-version-262689 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --container-runtime=containerd --kubernetes-version=v1.20.0
start_stop_delete_test.go:256: (dbg) Non-zero exit: out/minikube-linux-arm64 start -p old-k8s-version-262689 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --container-runtime=containerd --kubernetes-version=v1.20.0: exit status 102 (6m14.260094053s)
-- stdout --
* [old-k8s-version-262689] minikube v1.33.1 on Ubuntu 20.04 (arm64)
- MINIKUBE_LOCATION=19326
- MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
- KUBECONFIG=/home/jenkins/minikube-integration/19326-431487/kubeconfig
- MINIKUBE_HOME=/home/jenkins/minikube-integration/19326-431487/.minikube
- MINIKUBE_BIN=out/minikube-linux-arm64
- MINIKUBE_FORCE_SYSTEMD=
* Kubernetes 1.30.3 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.30.3
* Using the docker driver based on existing profile
* Starting "old-k8s-version-262689" primary control-plane node in "old-k8s-version-262689" cluster
* Pulling base image v0.0.44-1721902582-19326 ...
* Restarting existing docker container for "old-k8s-version-262689" ...
* Preparing Kubernetes v1.20.0 on containerd 1.7.19 ...
* Verifying Kubernetes components...
- Using image gcr.io/k8s-minikube/storage-provisioner:v5
- Using image docker.io/kubernetesui/dashboard:v2.7.0
- Using image fake.domain/registry.k8s.io/echoserver:1.4
- Using image registry.k8s.io/echoserver:1.4
* Some dashboard features require the metrics-server addon. To enable all features please run:
minikube -p old-k8s-version-262689 addons enable metrics-server
* Enabled addons: metrics-server, dashboard, storage-provisioner, default-storageclass
-- /stdout --
** stderr **
I0725 19:28:16.781932 688921 out.go:291] Setting OutFile to fd 1 ...
I0725 19:28:16.782122 688921 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0725 19:28:16.782151 688921 out.go:304] Setting ErrFile to fd 2...
I0725 19:28:16.782171 688921 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0725 19:28:16.782476 688921 root.go:338] Updating PATH: /home/jenkins/minikube-integration/19326-431487/.minikube/bin
I0725 19:28:16.782862 688921 out.go:298] Setting JSON to false
I0725 19:28:16.783949 688921 start.go:129] hostinfo: {"hostname":"ip-172-31-30-239","uptime":11446,"bootTime":1721924251,"procs":211,"os":"linux","platform":"ubuntu","platformFamily":"debian","platformVersion":"20.04","kernelVersion":"5.15.0-1065-aws","kernelArch":"aarch64","virtualizationSystem":"","virtualizationRole":"","hostId":"92f46a7d-c249-4c12-924a-77f64874c910"}
I0725 19:28:16.784048 688921 start.go:139] virtualization:
I0725 19:28:16.786539 688921 out.go:177] * [old-k8s-version-262689] minikube v1.33.1 on Ubuntu 20.04 (arm64)
I0725 19:28:16.789162 688921 out.go:177] - MINIKUBE_LOCATION=19326
I0725 19:28:16.789336 688921 notify.go:220] Checking for updates...
I0725 19:28:16.793393 688921 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0725 19:28:16.795348 688921 out.go:177] - KUBECONFIG=/home/jenkins/minikube-integration/19326-431487/kubeconfig
I0725 19:28:16.797182 688921 out.go:177] - MINIKUBE_HOME=/home/jenkins/minikube-integration/19326-431487/.minikube
I0725 19:28:16.799134 688921 out.go:177] - MINIKUBE_BIN=out/minikube-linux-arm64
I0725 19:28:16.801258 688921 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0725 19:28:16.803535 688921 config.go:182] Loaded profile config "old-k8s-version-262689": Driver=docker, ContainerRuntime=containerd, KubernetesVersion=v1.20.0
I0725 19:28:16.805808 688921 out.go:177] * Kubernetes 1.30.3 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.30.3
I0725 19:28:16.807557 688921 driver.go:392] Setting default libvirt URI to qemu:///system
I0725 19:28:16.840472 688921 docker.go:123] docker version: linux-27.1.1:Docker Engine - Community
I0725 19:28:16.840617 688921 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0725 19:28:16.899684 688921 info.go:266] docker info: {ID:6ZPO:QZND:VNGE:LUKL:4Y3K:XELL:AAX4:2GTK:E6LM:MPRN:3ZXR:TTMR Containers:2 ContainersRunning:1 ContainersPaused:0 ContainersStopped:1 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:39 OomKillDisable:true NGoroutines:51 SystemTime:2024-07-25 19:28:16.888866993 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1065-aws OperatingSystem:Ubuntu 20.04.6 LTS OSType:linux Architecture:aar
ch64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:2 MemTotal:8214900736 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ip-172-31-30-239 Labels:[] ExperimentalBuild:false ServerVersion:27.1.1 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41 Expected:2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41} RuncCommit:{ID:v1.1.13-0-g58aa920 Expected:v1.1.13-0-g58aa920} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=builtin] ProductLicense: Warnings:<nil> ServerErro
rs:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.16.1] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.29.1]] Warnings:<nil>}}
I0725 19:28:16.899810 688921 docker.go:307] overlay module found
I0725 19:28:16.903099 688921 out.go:177] * Using the docker driver based on existing profile
I0725 19:28:16.904738 688921 start.go:297] selected driver: docker
I0725 19:28:16.904757 688921 start.go:901] validating driver "docker" against &{Name:old-k8s-version-262689 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.0 ClusterName:old-k8s-version-262689 Namespace:default APIServerHAVIP: AP
IServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.85.2 Port:8443 KubernetesVersion:v1.20.0 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[dashboard:true default-storageclass:true metrics-server:true storage-provisioner:true] CustomAddonImages:map[MetricsScraper:registry.k8s.io/echoserver:1.4 MetricsServer:registry.k8s.io/echoserver:1.4] CustomAddonRegistries:map[MetricsServer:fake.domain] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:fa
lse MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0725 19:28:16.904867 688921 start.go:912] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0725 19:28:16.905536 688921 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0725 19:28:16.979605 688921 info.go:266] docker info: {ID:6ZPO:QZND:VNGE:LUKL:4Y3K:XELL:AAX4:2GTK:E6LM:MPRN:3ZXR:TTMR Containers:2 ContainersRunning:1 ContainersPaused:0 ContainersStopped:1 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:39 OomKillDisable:true NGoroutines:51 SystemTime:2024-07-25 19:28:16.970283159 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1065-aws OperatingSystem:Ubuntu 20.04.6 LTS OSType:linux Architecture:aar
ch64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:2 MemTotal:8214900736 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ip-172-31-30-239 Labels:[] ExperimentalBuild:false ServerVersion:27.1.1 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41 Expected:2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41} RuncCommit:{ID:v1.1.13-0-g58aa920 Expected:v1.1.13-0-g58aa920} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=builtin] ProductLicense: Warnings:<nil> ServerErro
rs:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.16.1] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.29.1]] Warnings:<nil>}}
I0725 19:28:16.980010 688921 start_flags.go:947] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0725 19:28:16.980058 688921 cni.go:84] Creating CNI manager for ""
I0725 19:28:16.980071 688921 cni.go:143] "docker" driver + "containerd" runtime found, recommending kindnet
I0725 19:28:16.980123 688921 start.go:340] cluster config:
{Name:old-k8s-version-262689 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.0 ClusterName:old-k8s-version-262689 Namespace:default APIServerHAVIP: APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local
ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.85.2 Port:8443 KubernetesVersion:v1.20.0 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[dashboard:true default-storageclass:true metrics-server:true storage-provisioner:true] CustomAddonImages:map[MetricsScraper:registry.k8s.io/echoserver:1.4 MetricsServer:registry.k8s.io/echoserver:1.4] CustomAddonRegistries:map[MetricsServer:fake.domain] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:do
cker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0725 19:28:16.982399 688921 out.go:177] * Starting "old-k8s-version-262689" primary control-plane node in "old-k8s-version-262689" cluster
I0725 19:28:16.984340 688921 cache.go:121] Beginning downloading kic base image for docker with containerd
I0725 19:28:16.986093 688921 out.go:177] * Pulling base image v0.0.44-1721902582-19326 ...
I0725 19:28:16.987887 688921 preload.go:131] Checking if preload exists for k8s version v1.20.0 and runtime containerd
I0725 19:28:16.987941 688921 preload.go:146] Found local preload: /home/jenkins/minikube-integration/19326-431487/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.20.0-containerd-overlay2-arm64.tar.lz4
I0725 19:28:16.987954 688921 cache.go:56] Caching tarball of preloaded images
I0725 19:28:16.987956 688921 image.go:79] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 in local docker daemon
I0725 19:28:16.988035 688921 preload.go:172] Found /home/jenkins/minikube-integration/19326-431487/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.20.0-containerd-overlay2-arm64.tar.lz4 in cache, skipping download
I0725 19:28:16.988044 688921 cache.go:59] Finished verifying existence of preloaded tar for v1.20.0 on containerd
I0725 19:28:16.988158 688921 profile.go:143] Saving config to /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/old-k8s-version-262689/config.json ...
W0725 19:28:17.011489 688921 image.go:95] image gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 is of wrong architecture
I0725 19:28:17.011511 688921 cache.go:149] Downloading gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 to local cache
I0725 19:28:17.011590 688921 image.go:63] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 in local cache directory
I0725 19:28:17.011621 688921 image.go:66] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 in local cache directory, skipping pull
I0725 19:28:17.011629 688921 image.go:135] gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 exists in cache, skipping pull
I0725 19:28:17.011638 688921 cache.go:152] successfully saved gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 as a tarball
I0725 19:28:17.011643 688921 cache.go:162] Loading gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 from local cache
I0725 19:28:17.198300 688921 cache.go:164] successfully loaded and using gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 from cached tarball
I0725 19:28:17.198380 688921 cache.go:194] Successfully downloaded all kic artifacts
I0725 19:28:17.198423 688921 start.go:360] acquireMachinesLock for old-k8s-version-262689: {Name:mk52d59fe6ca7e9ea9e4daeb7fea024956b4cbea Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0725 19:28:17.198511 688921 start.go:364] duration metric: took 55.309µs to acquireMachinesLock for "old-k8s-version-262689"
I0725 19:28:17.198536 688921 start.go:96] Skipping create...Using existing machine configuration
I0725 19:28:17.198545 688921 fix.go:54] fixHost starting:
I0725 19:28:17.198825 688921 cli_runner.go:164] Run: docker container inspect old-k8s-version-262689 --format={{.State.Status}}
I0725 19:28:17.215468 688921 fix.go:112] recreateIfNeeded on old-k8s-version-262689: state=Stopped err=<nil>
W0725 19:28:17.215506 688921 fix.go:138] unexpected machine state, will restart: <nil>
I0725 19:28:17.218201 688921 out.go:177] * Restarting existing docker container for "old-k8s-version-262689" ...
I0725 19:28:17.220270 688921 cli_runner.go:164] Run: docker start old-k8s-version-262689
I0725 19:28:17.531594 688921 cli_runner.go:164] Run: docker container inspect old-k8s-version-262689 --format={{.State.Status}}
I0725 19:28:17.558480 688921 kic.go:430] container "old-k8s-version-262689" state is running.
I0725 19:28:17.561171 688921 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-262689
I0725 19:28:17.584061 688921 profile.go:143] Saving config to /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/old-k8s-version-262689/config.json ...
I0725 19:28:17.584283 688921 machine.go:94] provisionDockerMachine start ...
I0725 19:28:17.584353 688921 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-262689
I0725 19:28:17.612330 688921 main.go:141] libmachine: Using SSH client type: native
I0725 19:28:17.612743 688921 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3e2cd0] 0x3e5530 <nil> [] 0s} 127.0.0.1 33496 <nil> <nil>}
I0725 19:28:17.612756 688921 main.go:141] libmachine: About to run SSH command:
hostname
I0725 19:28:17.613319 688921 main.go:141] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:58630->127.0.0.1:33496: read: connection reset by peer
I0725 19:28:20.778842 688921 main.go:141] libmachine: SSH cmd err, output: <nil>: old-k8s-version-262689
I0725 19:28:20.778875 688921 ubuntu.go:169] provisioning hostname "old-k8s-version-262689"
I0725 19:28:20.779003 688921 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-262689
I0725 19:28:20.800334 688921 main.go:141] libmachine: Using SSH client type: native
I0725 19:28:20.800596 688921 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3e2cd0] 0x3e5530 <nil> [] 0s} 127.0.0.1 33496 <nil> <nil>}
I0725 19:28:20.800613 688921 main.go:141] libmachine: About to run SSH command:
sudo hostname old-k8s-version-262689 && echo "old-k8s-version-262689" | sudo tee /etc/hostname
I0725 19:28:20.952370 688921 main.go:141] libmachine: SSH cmd err, output: <nil>: old-k8s-version-262689
I0725 19:28:20.952449 688921 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-262689
I0725 19:28:20.972122 688921 main.go:141] libmachine: Using SSH client type: native
I0725 19:28:20.972415 688921 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3e2cd0] 0x3e5530 <nil> [] 0s} 127.0.0.1 33496 <nil> <nil>}
I0725 19:28:20.972433 688921 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sold-k8s-version-262689' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 old-k8s-version-262689/g' /etc/hosts;
else
echo '127.0.1.1 old-k8s-version-262689' | sudo tee -a /etc/hosts;
fi
fi
I0725 19:28:21.119030 688921 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0725 19:28:21.119109 688921 ubuntu.go:175] set auth options {CertDir:/home/jenkins/minikube-integration/19326-431487/.minikube CaCertPath:/home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca.pem CaPrivateKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/server.pem ServerKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/server-key.pem ClientKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/home/jenkins/minikube-integration/19326-431487/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/home/jenkins/minikube-integration/19326-431487/.minikube}
I0725 19:28:21.119152 688921 ubuntu.go:177] setting up certificates
I0725 19:28:21.119163 688921 provision.go:84] configureAuth start
I0725 19:28:21.119231 688921 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-262689
I0725 19:28:21.136800 688921 provision.go:143] copyHostCerts
I0725 19:28:21.136873 688921 exec_runner.go:144] found /home/jenkins/minikube-integration/19326-431487/.minikube/ca.pem, removing ...
I0725 19:28:21.136890 688921 exec_runner.go:203] rm: /home/jenkins/minikube-integration/19326-431487/.minikube/ca.pem
I0725 19:28:21.136966 688921 exec_runner.go:151] cp: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca.pem --> /home/jenkins/minikube-integration/19326-431487/.minikube/ca.pem (1082 bytes)
I0725 19:28:21.137070 688921 exec_runner.go:144] found /home/jenkins/minikube-integration/19326-431487/.minikube/cert.pem, removing ...
I0725 19:28:21.137080 688921 exec_runner.go:203] rm: /home/jenkins/minikube-integration/19326-431487/.minikube/cert.pem
I0725 19:28:21.137106 688921 exec_runner.go:151] cp: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/cert.pem --> /home/jenkins/minikube-integration/19326-431487/.minikube/cert.pem (1123 bytes)
I0725 19:28:21.137164 688921 exec_runner.go:144] found /home/jenkins/minikube-integration/19326-431487/.minikube/key.pem, removing ...
I0725 19:28:21.137173 688921 exec_runner.go:203] rm: /home/jenkins/minikube-integration/19326-431487/.minikube/key.pem
I0725 19:28:21.137197 688921 exec_runner.go:151] cp: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/key.pem --> /home/jenkins/minikube-integration/19326-431487/.minikube/key.pem (1679 bytes)
I0725 19:28:21.137250 688921 provision.go:117] generating server cert: /home/jenkins/minikube-integration/19326-431487/.minikube/machines/server.pem ca-key=/home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca.pem private-key=/home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca-key.pem org=jenkins.old-k8s-version-262689 san=[127.0.0.1 192.168.85.2 localhost minikube old-k8s-version-262689]
I0725 19:28:21.556411 688921 provision.go:177] copyRemoteCerts
I0725 19:28:21.556483 688921 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0725 19:28:21.556532 688921 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-262689
I0725 19:28:21.574810 688921 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33496 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/old-k8s-version-262689/id_rsa Username:docker}
I0725 19:28:21.672235 688921 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1082 bytes)
I0725 19:28:21.697894 688921 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/machines/server.pem --> /etc/docker/server.pem (1233 bytes)
I0725 19:28:21.722590 688921 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1675 bytes)
I0725 19:28:21.748096 688921 provision.go:87] duration metric: took 628.919694ms to configureAuth
I0725 19:28:21.748123 688921 ubuntu.go:193] setting minikube options for container-runtime
I0725 19:28:21.748328 688921 config.go:182] Loaded profile config "old-k8s-version-262689": Driver=docker, ContainerRuntime=containerd, KubernetesVersion=v1.20.0
I0725 19:28:21.748335 688921 machine.go:97] duration metric: took 4.164044592s to provisionDockerMachine
I0725 19:28:21.748343 688921 start.go:293] postStartSetup for "old-k8s-version-262689" (driver="docker")
I0725 19:28:21.748354 688921 start.go:322] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0725 19:28:21.748408 688921 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0725 19:28:21.748447 688921 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-262689
I0725 19:28:21.765175 688921 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33496 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/old-k8s-version-262689/id_rsa Username:docker}
I0725 19:28:21.860149 688921 ssh_runner.go:195] Run: cat /etc/os-release
I0725 19:28:21.863462 688921 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0725 19:28:21.863497 688921 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0725 19:28:21.863508 688921 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0725 19:28:21.863516 688921 info.go:137] Remote host: Ubuntu 22.04.4 LTS
I0725 19:28:21.863529 688921 filesync.go:126] Scanning /home/jenkins/minikube-integration/19326-431487/.minikube/addons for local assets ...
I0725 19:28:21.863598 688921 filesync.go:126] Scanning /home/jenkins/minikube-integration/19326-431487/.minikube/files for local assets ...
I0725 19:28:21.863689 688921 filesync.go:149] local asset: /home/jenkins/minikube-integration/19326-431487/.minikube/files/etc/ssl/certs/4368932.pem -> 4368932.pem in /etc/ssl/certs
I0725 19:28:21.863798 688921 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0725 19:28:21.872378 688921 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/files/etc/ssl/certs/4368932.pem --> /etc/ssl/certs/4368932.pem (1708 bytes)
I0725 19:28:21.897064 688921 start.go:296] duration metric: took 148.705696ms for postStartSetup
I0725 19:28:21.897162 688921 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0725 19:28:21.897213 688921 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-262689
I0725 19:28:21.914083 688921 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33496 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/old-k8s-version-262689/id_rsa Username:docker}
I0725 19:28:22.005476 688921 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0725 19:28:22.011775 688921 fix.go:56] duration metric: took 4.813219299s for fixHost
I0725 19:28:22.011805 688921 start.go:83] releasing machines lock for "old-k8s-version-262689", held for 4.813282379s
I0725 19:28:22.011888 688921 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-262689
I0725 19:28:22.029451 688921 ssh_runner.go:195] Run: cat /version.json
I0725 19:28:22.029523 688921 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-262689
I0725 19:28:22.029800 688921 ssh_runner.go:195] Run: curl -sS -m 2 https://registry.k8s.io/
I0725 19:28:22.029876 688921 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-262689
I0725 19:28:22.051181 688921 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33496 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/old-k8s-version-262689/id_rsa Username:docker}
I0725 19:28:22.055319 688921 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33496 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/old-k8s-version-262689/id_rsa Username:docker}
I0725 19:28:22.279939 688921 ssh_runner.go:195] Run: systemctl --version
I0725 19:28:22.284521 688921 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
I0725 19:28:22.288879 688921 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *loopback.conf* -not -name *.mk_disabled -exec sh -c "grep -q loopback {} && ( grep -q name {} || sudo sed -i '/"type": "loopback"/i \ \ \ \ "name": "loopback",' {} ) && sudo sed -i 's|"cniVersion": ".*"|"cniVersion": "1.0.0"|g' {}" ;
I0725 19:28:22.307468 688921 cni.go:230] loopback cni configuration patched: "/etc/cni/net.d/*loopback.conf*" found
I0725 19:28:22.307577 688921 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f ( ( -name *bridge* -or -name *podman* ) -and -not -name *.mk_disabled ) -printf "%p, " -exec sh -c "sudo mv {} {}.mk_disabled" ;
I0725 19:28:22.317035 688921 cni.go:259] no active bridge cni configs found in "/etc/cni/net.d" - nothing to disable
I0725 19:28:22.317071 688921 start.go:495] detecting cgroup driver to use...
I0725 19:28:22.317104 688921 detect.go:187] detected "cgroupfs" cgroup driver on host os
I0725 19:28:22.317155 688921 ssh_runner.go:195] Run: sudo systemctl stop -f crio
I0725 19:28:22.331072 688921 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0725 19:28:22.343398 688921 docker.go:217] disabling cri-docker service (if available) ...
I0725 19:28:22.343489 688921 ssh_runner.go:195] Run: sudo systemctl stop -f cri-docker.socket
I0725 19:28:22.357167 688921 ssh_runner.go:195] Run: sudo systemctl stop -f cri-docker.service
I0725 19:28:22.368811 688921 ssh_runner.go:195] Run: sudo systemctl disable cri-docker.socket
I0725 19:28:22.462260 688921 ssh_runner.go:195] Run: sudo systemctl mask cri-docker.service
I0725 19:28:22.544348 688921 docker.go:233] disabling docker service ...
I0725 19:28:22.544428 688921 ssh_runner.go:195] Run: sudo systemctl stop -f docker.socket
I0725 19:28:22.556764 688921 ssh_runner.go:195] Run: sudo systemctl stop -f docker.service
I0725 19:28:22.568299 688921 ssh_runner.go:195] Run: sudo systemctl disable docker.socket
I0725 19:28:22.647176 688921 ssh_runner.go:195] Run: sudo systemctl mask docker.service
I0725 19:28:22.728262 688921 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service docker
I0725 19:28:22.740212 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0725 19:28:22.757196 688921 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.2"|' /etc/containerd/config.toml"
I0725 19:28:22.768753 688921 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I0725 19:28:22.778693 688921 containerd.go:146] configuring containerd to use "cgroupfs" as cgroup driver...
I0725 19:28:22.778779 688921 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = false|g' /etc/containerd/config.toml"
I0725 19:28:22.789367 688921 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0725 19:28:22.799803 688921 ssh_runner.go:195] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I0725 19:28:22.809741 688921 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0725 19:28:22.819795 688921 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0725 19:28:22.829369 688921 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I0725 19:28:22.839530 688921 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0725 19:28:22.848357 688921 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0725 19:28:22.856953 688921 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0725 19:28:22.943881 688921 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0725 19:28:23.120624 688921 start.go:542] Will wait 60s for socket path /run/containerd/containerd.sock
I0725 19:28:23.120708 688921 ssh_runner.go:195] Run: stat /run/containerd/containerd.sock
I0725 19:28:23.125252 688921 start.go:563] Will wait 60s for crictl version
I0725 19:28:23.125332 688921 ssh_runner.go:195] Run: which crictl
I0725 19:28:23.129711 688921 ssh_runner.go:195] Run: sudo /usr/bin/crictl version
I0725 19:28:23.170378 688921 start.go:579] Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: 1.7.19
RuntimeApiVersion: v1
I0725 19:28:23.170480 688921 ssh_runner.go:195] Run: containerd --version
I0725 19:28:23.196798 688921 ssh_runner.go:195] Run: containerd --version
I0725 19:28:23.226703 688921 out.go:177] * Preparing Kubernetes v1.20.0 on containerd 1.7.19 ...
I0725 19:28:23.228377 688921 cli_runner.go:164] Run: docker network inspect old-k8s-version-262689 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0725 19:28:23.244122 688921 ssh_runner.go:195] Run: grep 192.168.85.1 host.minikube.internal$ /etc/hosts
I0725 19:28:23.247719 688921 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.85.1 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0725 19:28:23.258886 688921 kubeadm.go:883] updating cluster {Name:old-k8s-version-262689 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.0 ClusterName:old-k8s-version-262689 Namespace:default APIServerHAVIP: APIServerName:minik
ubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.85.2 Port:8443 KubernetesVersion:v1.20.0 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[dashboard:true default-storageclass:true metrics-server:true storage-provisioner:true] CustomAddonImages:map[MetricsScraper:registry.k8s.io/echoserver:1.4 MetricsServer:registry.k8s.io/echoserver:1.4] CustomAddonRegistries:map[MetricsServer:fake.domain] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/
home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s} ...
I0725 19:28:23.259036 688921 preload.go:131] Checking if preload exists for k8s version v1.20.0 and runtime containerd
I0725 19:28:23.259099 688921 ssh_runner.go:195] Run: sudo crictl images --output json
I0725 19:28:23.297863 688921 containerd.go:627] all images are preloaded for containerd runtime.
I0725 19:28:23.297889 688921 containerd.go:534] Images already preloaded, skipping extraction
I0725 19:28:23.297951 688921 ssh_runner.go:195] Run: sudo crictl images --output json
I0725 19:28:23.334709 688921 containerd.go:627] all images are preloaded for containerd runtime.
I0725 19:28:23.334732 688921 cache_images.go:84] Images are preloaded, skipping loading
I0725 19:28:23.334741 688921 kubeadm.go:934] updating node { 192.168.85.2 8443 v1.20.0 containerd true true} ...
I0725 19:28:23.334854 688921 kubeadm.go:946] kubelet [Unit]
Wants=containerd.service
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.20.0/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --hostname-override=old-k8s-version-262689 --kubeconfig=/etc/kubernetes/kubelet.conf --network-plugin=cni --node-ip=192.168.85.2
[Install]
config:
{KubernetesVersion:v1.20.0 ClusterName:old-k8s-version-262689 Namespace:default APIServerHAVIP: APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:}
I0725 19:28:23.334965 688921 ssh_runner.go:195] Run: sudo crictl info
I0725 19:28:23.377347 688921 cni.go:84] Creating CNI manager for ""
I0725 19:28:23.377376 688921 cni.go:143] "docker" driver + "containerd" runtime found, recommending kindnet
I0725 19:28:23.377390 688921 kubeadm.go:84] Using pod CIDR: 10.244.0.0/16
I0725 19:28:23.377414 688921 kubeadm.go:181] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.85.2 APIServerPort:8443 KubernetesVersion:v1.20.0 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:old-k8s-version-262689 NodeName:old-k8s-version-262689 DNSDomain:cluster.local CRISocket:/run/containerd/containerd.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.85.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.85.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt
StaticPodPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[hairpinMode:hairpin-veth runtimeRequestTimeout:15m] PrependCriSocketUnix:false}
I0725 19:28:23.377550 688921 kubeadm.go:187] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.85.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: /run/containerd/containerd.sock
name: "old-k8s-version-262689"
kubeletExtraArgs:
node-ip: 192.168.85.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.85.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: mk
controlPlaneEndpoint: control-plane.minikube.internal:8443
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
proxy-refresh-interval: "70000"
kubernetesVersion: v1.20.0
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0725 19:28:23.377626 688921 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.20.0
I0725 19:28:23.387456 688921 binaries.go:44] Found k8s binaries, skipping transfer
I0725 19:28:23.387588 688921 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0725 19:28:23.396709 688921 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (442 bytes)
I0725 19:28:23.416069 688921 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0725 19:28:23.434474 688921 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2125 bytes)
I0725 19:28:23.453325 688921 ssh_runner.go:195] Run: grep 192.168.85.2 control-plane.minikube.internal$ /etc/hosts
I0725 19:28:23.456851 688921 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.85.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0725 19:28:23.468292 688921 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0725 19:28:23.561090 688921 ssh_runner.go:195] Run: sudo systemctl start kubelet
I0725 19:28:23.575265 688921 certs.go:68] Setting up /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/old-k8s-version-262689 for IP: 192.168.85.2
I0725 19:28:23.575329 688921 certs.go:194] generating shared ca certs ...
I0725 19:28:23.575379 688921 certs.go:226] acquiring lock for ca certs: {Name:mk41d7b1e7cb52699a093c81e00768f54d73ad8a Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0725 19:28:23.575587 688921 certs.go:235] skipping valid "minikubeCA" ca cert: /home/jenkins/minikube-integration/19326-431487/.minikube/ca.key
I0725 19:28:23.575680 688921 certs.go:235] skipping valid "proxyClientCA" ca cert: /home/jenkins/minikube-integration/19326-431487/.minikube/proxy-client-ca.key
I0725 19:28:23.575713 688921 certs.go:256] generating profile certs ...
I0725 19:28:23.575834 688921 certs.go:359] skipping valid signed profile cert regeneration for "minikube-user": /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/old-k8s-version-262689/client.key
I0725 19:28:23.576632 688921 certs.go:359] skipping valid signed profile cert regeneration for "minikube": /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/old-k8s-version-262689/apiserver.key.bc2b7a55
I0725 19:28:23.577315 688921 certs.go:359] skipping valid signed profile cert regeneration for "aggregator": /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/old-k8s-version-262689/proxy-client.key
I0725 19:28:23.577498 688921 certs.go:484] found cert: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/436893.pem (1338 bytes)
W0725 19:28:23.577597 688921 certs.go:480] ignoring /home/jenkins/minikube-integration/19326-431487/.minikube/certs/436893_empty.pem, impossibly tiny 0 bytes
I0725 19:28:23.577636 688921 certs.go:484] found cert: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca-key.pem (1675 bytes)
I0725 19:28:23.577693 688921 certs.go:484] found cert: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca.pem (1082 bytes)
I0725 19:28:23.577744 688921 certs.go:484] found cert: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/cert.pem (1123 bytes)
I0725 19:28:23.577797 688921 certs.go:484] found cert: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/key.pem (1679 bytes)
I0725 19:28:23.577872 688921 certs.go:484] found cert: /home/jenkins/minikube-integration/19326-431487/.minikube/files/etc/ssl/certs/4368932.pem (1708 bytes)
I0725 19:28:23.578548 688921 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0725 19:28:23.613381 688921 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1679 bytes)
I0725 19:28:23.638752 688921 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0725 19:28:23.668297 688921 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1675 bytes)
I0725 19:28:23.697194 688921 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/old-k8s-version-262689/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1432 bytes)
I0725 19:28:23.728342 688921 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/old-k8s-version-262689/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1679 bytes)
I0725 19:28:23.758289 688921 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/old-k8s-version-262689/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0725 19:28:23.790378 688921 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/old-k8s-version-262689/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1679 bytes)
I0725 19:28:23.816130 688921 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/files/etc/ssl/certs/4368932.pem --> /usr/share/ca-certificates/4368932.pem (1708 bytes)
I0725 19:28:23.841445 688921 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0725 19:28:23.867182 688921 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/certs/436893.pem --> /usr/share/ca-certificates/436893.pem (1338 bytes)
I0725 19:28:23.891951 688921 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0725 19:28:23.910425 688921 ssh_runner.go:195] Run: openssl version
I0725 19:28:23.915944 688921 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/4368932.pem && ln -fs /usr/share/ca-certificates/4368932.pem /etc/ssl/certs/4368932.pem"
I0725 19:28:23.925736 688921 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/4368932.pem
I0725 19:28:23.929384 688921 certs.go:528] hashing: -rw-r--r-- 1 root root 1708 Jul 25 18:40 /usr/share/ca-certificates/4368932.pem
I0725 19:28:23.929470 688921 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/4368932.pem
I0725 19:28:23.936486 688921 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/4368932.pem /etc/ssl/certs/3ec20f2e.0"
I0725 19:28:23.945613 688921 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0725 19:28:23.955221 688921 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0725 19:28:23.958826 688921 certs.go:528] hashing: -rw-r--r-- 1 root root 1111 Jul 25 18:30 /usr/share/ca-certificates/minikubeCA.pem
I0725 19:28:23.959032 688921 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0725 19:28:23.966322 688921 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0725 19:28:23.975731 688921 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/436893.pem && ln -fs /usr/share/ca-certificates/436893.pem /etc/ssl/certs/436893.pem"
I0725 19:28:23.985325 688921 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/436893.pem
I0725 19:28:23.988971 688921 certs.go:528] hashing: -rw-r--r-- 1 root root 1338 Jul 25 18:40 /usr/share/ca-certificates/436893.pem
I0725 19:28:23.989062 688921 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/436893.pem
I0725 19:28:23.996226 688921 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/436893.pem /etc/ssl/certs/51391683.0"
I0725 19:28:24.007920 688921 ssh_runner.go:195] Run: stat /var/lib/minikube/certs/apiserver-kubelet-client.crt
I0725 19:28:24.013285 688921 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/apiserver-etcd-client.crt -checkend 86400
I0725 19:28:24.021426 688921 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/apiserver-kubelet-client.crt -checkend 86400
I0725 19:28:24.029383 688921 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/etcd/server.crt -checkend 86400
I0725 19:28:24.037453 688921 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/etcd/healthcheck-client.crt -checkend 86400
I0725 19:28:24.045450 688921 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/etcd/peer.crt -checkend 86400
I0725 19:28:24.053091 688921 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/front-proxy-client.crt -checkend 86400
I0725 19:28:24.060745 688921 kubeadm.go:392] StartCluster: {Name:old-k8s-version-262689 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.0 ClusterName:old-k8s-version-262689 Namespace:default APIServerHAVIP: APIServerName:minikube
CA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.85.2 Port:8443 KubernetesVersion:v1.20.0 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[dashboard:true default-storageclass:true metrics-server:true storage-provisioner:true] CustomAddonImages:map[MetricsScraper:registry.k8s.io/echoserver:1.4 MetricsServer:registry.k8s.io/echoserver:1.4] CustomAddonRegistries:map[MetricsServer:fake.domain] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/hom
e/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0725 19:28:24.060849 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:paused Name: Namespaces:[kube-system]}
I0725 19:28:24.060928 688921 ssh_runner.go:195] Run: sudo -s eval "crictl ps -a --quiet --label io.kubernetes.pod.namespace=kube-system"
I0725 19:28:24.118182 688921 cri.go:89] found id: "6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8"
I0725 19:28:24.118212 688921 cri.go:89] found id: "687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8"
I0725 19:28:24.118217 688921 cri.go:89] found id: "1d9b398db1989c14a18312ca23045842653811f0be4341afde62a30b1e42aa41"
I0725 19:28:24.118221 688921 cri.go:89] found id: "5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f"
I0725 19:28:24.118224 688921 cri.go:89] found id: "2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34"
I0725 19:28:24.118227 688921 cri.go:89] found id: "092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb"
I0725 19:28:24.118231 688921 cri.go:89] found id: "a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8"
I0725 19:28:24.118234 688921 cri.go:89] found id: "bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6"
I0725 19:28:24.118237 688921 cri.go:89] found id: ""
I0725 19:28:24.118298 688921 ssh_runner.go:195] Run: sudo runc --root /run/containerd/runc/k8s.io list -f json
I0725 19:28:24.131472 688921 cri.go:116] JSON = null
W0725 19:28:24.131574 688921 kubeadm.go:399] unpause failed: list paused: list returned 0 containers, but ps returned 8
I0725 19:28:24.131678 688921 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0725 19:28:24.141383 688921 kubeadm.go:408] found existing configuration files, will attempt cluster restart
I0725 19:28:24.141404 688921 kubeadm.go:593] restartPrimaryControlPlane start ...
I0725 19:28:24.141484 688921 ssh_runner.go:195] Run: sudo test -d /data/minikube
I0725 19:28:24.151055 688921 kubeadm.go:130] /data/minikube skipping compat symlinks: sudo test -d /data/minikube: Process exited with status 1
stdout:
stderr:
I0725 19:28:24.151747 688921 kubeconfig.go:47] verify endpoint returned: get endpoint: "old-k8s-version-262689" does not appear in /home/jenkins/minikube-integration/19326-431487/kubeconfig
I0725 19:28:24.152015 688921 kubeconfig.go:62] /home/jenkins/minikube-integration/19326-431487/kubeconfig needs updating (will repair): [kubeconfig missing "old-k8s-version-262689" cluster setting kubeconfig missing "old-k8s-version-262689" context setting]
I0725 19:28:24.152551 688921 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/19326-431487/kubeconfig: {Name:mk3cdfe1101bbbc0f7441d92cff5cd6b29ee3404 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0725 19:28:24.153979 688921 ssh_runner.go:195] Run: sudo diff -u /var/tmp/minikube/kubeadm.yaml /var/tmp/minikube/kubeadm.yaml.new
I0725 19:28:24.165735 688921 kubeadm.go:630] The running cluster does not require reconfiguration: 192.168.85.2
I0725 19:28:24.165767 688921 kubeadm.go:597] duration metric: took 24.356597ms to restartPrimaryControlPlane
I0725 19:28:24.165776 688921 kubeadm.go:394] duration metric: took 105.04483ms to StartCluster
I0725 19:28:24.165791 688921 settings.go:142] acquiring lock: {Name:mk69edff96840eebb76289a50cf78daf601fe5de Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0725 19:28:24.165849 688921 settings.go:150] Updating kubeconfig: /home/jenkins/minikube-integration/19326-431487/kubeconfig
I0725 19:28:24.166744 688921 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/19326-431487/kubeconfig: {Name:mk3cdfe1101bbbc0f7441d92cff5cd6b29ee3404 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0725 19:28:24.167046 688921 start.go:235] Will wait 6m0s for node &{Name: IP:192.168.85.2 Port:8443 KubernetesVersion:v1.20.0 ContainerRuntime:containerd ControlPlane:true Worker:true}
I0725 19:28:24.167322 688921 config.go:182] Loaded profile config "old-k8s-version-262689": Driver=docker, ContainerRuntime=containerd, KubernetesVersion=v1.20.0
I0725 19:28:24.167399 688921 addons.go:507] enable addons start: toEnable=map[ambassador:false auto-pause:false cloud-spanner:false csi-hostpath-driver:false dashboard:true default-storageclass:true efk:false freshpod:false gcp-auth:false gvisor:false headlamp:false helm-tiller:false inaccel:false ingress:false ingress-dns:false inspektor-gadget:false istio:false istio-provisioner:false kong:false kubeflow:false kubevirt:false logviewer:false metallb:false metrics-server:true nvidia-device-plugin:false nvidia-driver-installer:false nvidia-gpu-device-plugin:false olm:false pod-security-policy:false portainer:false registry:false registry-aliases:false registry-creds:false storage-provisioner:true storage-provisioner-gluster:false storage-provisioner-rancher:false volcano:false volumesnapshots:false yakd:false]
I0725 19:28:24.167545 688921 addons.go:69] Setting storage-provisioner=true in profile "old-k8s-version-262689"
I0725 19:28:24.167585 688921 addons.go:234] Setting addon storage-provisioner=true in "old-k8s-version-262689"
W0725 19:28:24.167659 688921 addons.go:243] addon storage-provisioner should already be in state true
I0725 19:28:24.167687 688921 host.go:66] Checking if "old-k8s-version-262689" exists ...
I0725 19:28:24.167619 688921 addons.go:69] Setting default-storageclass=true in profile "old-k8s-version-262689"
I0725 19:28:24.167773 688921 addons_storage_classes.go:33] enableOrDisableStorageClasses default-storageclass=true on "old-k8s-version-262689"
I0725 19:28:24.168091 688921 cli_runner.go:164] Run: docker container inspect old-k8s-version-262689 --format={{.State.Status}}
I0725 19:28:24.168109 688921 cli_runner.go:164] Run: docker container inspect old-k8s-version-262689 --format={{.State.Status}}
I0725 19:28:24.167624 688921 addons.go:69] Setting dashboard=true in profile "old-k8s-version-262689"
I0725 19:28:24.168577 688921 addons.go:234] Setting addon dashboard=true in "old-k8s-version-262689"
W0725 19:28:24.168592 688921 addons.go:243] addon dashboard should already be in state true
I0725 19:28:24.168624 688921 host.go:66] Checking if "old-k8s-version-262689" exists ...
I0725 19:28:24.169026 688921 cli_runner.go:164] Run: docker container inspect old-k8s-version-262689 --format={{.State.Status}}
I0725 19:28:24.172116 688921 out.go:177] * Verifying Kubernetes components...
I0725 19:28:24.167629 688921 addons.go:69] Setting metrics-server=true in profile "old-k8s-version-262689"
I0725 19:28:24.172531 688921 addons.go:234] Setting addon metrics-server=true in "old-k8s-version-262689"
W0725 19:28:24.172546 688921 addons.go:243] addon metrics-server should already be in state true
I0725 19:28:24.172579 688921 host.go:66] Checking if "old-k8s-version-262689" exists ...
I0725 19:28:24.172991 688921 cli_runner.go:164] Run: docker container inspect old-k8s-version-262689 --format={{.State.Status}}
I0725 19:28:24.180683 688921 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0725 19:28:24.197765 688921 addons.go:234] Setting addon default-storageclass=true in "old-k8s-version-262689"
W0725 19:28:24.197789 688921 addons.go:243] addon default-storageclass should already be in state true
I0725 19:28:24.197815 688921 host.go:66] Checking if "old-k8s-version-262689" exists ...
I0725 19:28:24.198200 688921 cli_runner.go:164] Run: docker container inspect old-k8s-version-262689 --format={{.State.Status}}
I0725 19:28:24.222601 688921 addons.go:431] installing /etc/kubernetes/addons/storageclass.yaml
I0725 19:28:24.222628 688921 ssh_runner.go:362] scp storageclass/storageclass.yaml --> /etc/kubernetes/addons/storageclass.yaml (271 bytes)
I0725 19:28:24.222695 688921 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-262689
I0725 19:28:24.225684 688921 out.go:177] - Using image gcr.io/k8s-minikube/storage-provisioner:v5
I0725 19:28:24.227599 688921 addons.go:431] installing /etc/kubernetes/addons/storage-provisioner.yaml
I0725 19:28:24.227619 688921 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/storage-provisioner.yaml (2676 bytes)
I0725 19:28:24.227688 688921 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-262689
I0725 19:28:24.253248 688921 out.go:177] - Using image docker.io/kubernetesui/dashboard:v2.7.0
I0725 19:28:24.253247 688921 out.go:177] - Using image fake.domain/registry.k8s.io/echoserver:1.4
I0725 19:28:24.255691 688921 addons.go:431] installing /etc/kubernetes/addons/metrics-apiservice.yaml
I0725 19:28:24.255721 688921 ssh_runner.go:362] scp metrics-server/metrics-apiservice.yaml --> /etc/kubernetes/addons/metrics-apiservice.yaml (424 bytes)
I0725 19:28:24.255798 688921 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-262689
I0725 19:28:24.258004 688921 out.go:177] - Using image registry.k8s.io/echoserver:1.4
I0725 19:28:24.260144 688921 addons.go:431] installing /etc/kubernetes/addons/dashboard-ns.yaml
I0725 19:28:24.260170 688921 ssh_runner.go:362] scp dashboard/dashboard-ns.yaml --> /etc/kubernetes/addons/dashboard-ns.yaml (759 bytes)
I0725 19:28:24.260239 688921 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-262689
I0725 19:28:24.304989 688921 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33496 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/old-k8s-version-262689/id_rsa Username:docker}
I0725 19:28:24.305478 688921 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33496 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/old-k8s-version-262689/id_rsa Username:docker}
I0725 19:28:24.313991 688921 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33496 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/old-k8s-version-262689/id_rsa Username:docker}
I0725 19:28:24.329077 688921 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33496 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/old-k8s-version-262689/id_rsa Username:docker}
I0725 19:28:24.340109 688921 ssh_runner.go:195] Run: sudo systemctl start kubelet
I0725 19:28:24.360695 688921 node_ready.go:35] waiting up to 6m0s for node "old-k8s-version-262689" to be "Ready" ...
I0725 19:28:24.445495 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/storageclass.yaml
I0725 19:28:24.453347 688921 addons.go:431] installing /etc/kubernetes/addons/metrics-server-deployment.yaml
I0725 19:28:24.453372 688921 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/metrics-server-deployment.yaml (1825 bytes)
I0725 19:28:24.485068 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/storage-provisioner.yaml
I0725 19:28:24.490093 688921 addons.go:431] installing /etc/kubernetes/addons/metrics-server-rbac.yaml
I0725 19:28:24.490115 688921 ssh_runner.go:362] scp metrics-server/metrics-server-rbac.yaml --> /etc/kubernetes/addons/metrics-server-rbac.yaml (2175 bytes)
I0725 19:28:24.502415 688921 addons.go:431] installing /etc/kubernetes/addons/dashboard-clusterrole.yaml
I0725 19:28:24.502438 688921 ssh_runner.go:362] scp dashboard/dashboard-clusterrole.yaml --> /etc/kubernetes/addons/dashboard-clusterrole.yaml (1001 bytes)
I0725 19:28:24.564168 688921 addons.go:431] installing /etc/kubernetes/addons/metrics-server-service.yaml
I0725 19:28:24.564190 688921 ssh_runner.go:362] scp metrics-server/metrics-server-service.yaml --> /etc/kubernetes/addons/metrics-server-service.yaml (446 bytes)
I0725 19:28:24.566501 688921 addons.go:431] installing /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml
I0725 19:28:24.566521 688921 ssh_runner.go:362] scp dashboard/dashboard-clusterrolebinding.yaml --> /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml (1018 bytes)
I0725 19:28:24.605415 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
I0725 19:28:24.607905 688921 addons.go:431] installing /etc/kubernetes/addons/dashboard-configmap.yaml
I0725 19:28:24.607926 688921 ssh_runner.go:362] scp dashboard/dashboard-configmap.yaml --> /etc/kubernetes/addons/dashboard-configmap.yaml (837 bytes)
W0725 19:28:24.630747 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:24.630782 688921 retry.go:31] will retry after 283.280345ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
W0725 19:28:24.650092 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:24.650125 688921 retry.go:31] will retry after 192.18969ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:24.660461 688921 addons.go:431] installing /etc/kubernetes/addons/dashboard-dp.yaml
I0725 19:28:24.660488 688921 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/dashboard-dp.yaml (4201 bytes)
I0725 19:28:24.679476 688921 addons.go:431] installing /etc/kubernetes/addons/dashboard-role.yaml
I0725 19:28:24.679502 688921 ssh_runner.go:362] scp dashboard/dashboard-role.yaml --> /etc/kubernetes/addons/dashboard-role.yaml (1724 bytes)
I0725 19:28:24.697789 688921 addons.go:431] installing /etc/kubernetes/addons/dashboard-rolebinding.yaml
I0725 19:28:24.697811 688921 ssh_runner.go:362] scp dashboard/dashboard-rolebinding.yaml --> /etc/kubernetes/addons/dashboard-rolebinding.yaml (1046 bytes)
I0725 19:28:24.715737 688921 addons.go:431] installing /etc/kubernetes/addons/dashboard-sa.yaml
I0725 19:28:24.715759 688921 ssh_runner.go:362] scp dashboard/dashboard-sa.yaml --> /etc/kubernetes/addons/dashboard-sa.yaml (837 bytes)
I0725 19:28:24.734059 688921 addons.go:431] installing /etc/kubernetes/addons/dashboard-secret.yaml
I0725 19:28:24.734082 688921 ssh_runner.go:362] scp dashboard/dashboard-secret.yaml --> /etc/kubernetes/addons/dashboard-secret.yaml (1389 bytes)
W0725 19:28:24.745766 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:24.745798 688921 retry.go:31] will retry after 280.658949ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:24.753238 688921 addons.go:431] installing /etc/kubernetes/addons/dashboard-svc.yaml
I0725 19:28:24.753309 688921 ssh_runner.go:362] scp dashboard/dashboard-svc.yaml --> /etc/kubernetes/addons/dashboard-svc.yaml (1294 bytes)
I0725 19:28:24.773744 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
W0725 19:28:24.842425 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:24.842500 688921 retry.go:31] will retry after 302.32861ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:24.842434 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml
I0725 19:28:24.914535 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml
W0725 19:28:24.921810 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:24.921883 688921 retry.go:31] will retry after 234.409766ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
W0725 19:28:24.991177 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:24.991210 688921 retry.go:31] will retry after 445.227503ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:25.027465 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
W0725 19:28:25.104088 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:25.104132 688921 retry.go:31] will retry after 533.613917ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:25.145375 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
I0725 19:28:25.156749 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml
W0725 19:28:25.241267 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:25.241313 688921 retry.go:31] will retry after 313.63186ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
W0725 19:28:25.266537 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:25.266569 688921 retry.go:31] will retry after 460.201706ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:25.437549 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml
W0725 19:28:25.515571 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:25.515604 688921 retry.go:31] will retry after 755.935876ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:25.555800 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
W0725 19:28:25.625591 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:25.625623 688921 retry.go:31] will retry after 785.377318ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:25.638743 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
W0725 19:28:25.707428 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:25.707460 688921 retry.go:31] will retry after 794.102193ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:25.727779 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml
W0725 19:28:25.799931 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:25.799961 688921 retry.go:31] will retry after 973.230313ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:26.272278 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml
W0725 19:28:26.347605 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:26.347678 688921 retry.go:31] will retry after 1.044600994s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:26.362263 688921 node_ready.go:53] error getting node "old-k8s-version-262689": Get "https://192.168.85.2:8443/api/v1/nodes/old-k8s-version-262689": dial tcp 192.168.85.2:8443: connect: connection refused
I0725 19:28:26.411633 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
W0725 19:28:26.485918 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:26.485958 688921 retry.go:31] will retry after 605.710532ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:26.502261 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
W0725 19:28:26.578560 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:26.578612 688921 retry.go:31] will retry after 1.152231704s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:26.774379 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml
W0725 19:28:26.844288 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:26.844325 688921 retry.go:31] will retry after 1.032868324s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:27.092646 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
W0725 19:28:27.163796 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:27.163832 688921 retry.go:31] will retry after 1.499808843s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:27.392679 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml
W0725 19:28:27.464917 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:27.464947 688921 retry.go:31] will retry after 1.625870195s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:27.731183 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
W0725 19:28:27.802896 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:27.802951 688921 retry.go:31] will retry after 973.836717ms: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:27.877849 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml
W0725 19:28:27.952942 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:27.953029 688921 retry.go:31] will retry after 1.860552888s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:28.664560 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
W0725 19:28:28.736880 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:28.736912 688921 retry.go:31] will retry after 1.347570963s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:28.777014 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
W0725 19:28:28.850737 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:28.850770 688921 retry.go:31] will retry after 2.379555503s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:28.861440 688921 node_ready.go:53] error getting node "old-k8s-version-262689": Get "https://192.168.85.2:8443/api/v1/nodes/old-k8s-version-262689": dial tcp 192.168.85.2:8443: connect: connection refused
I0725 19:28:29.091855 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml
W0725 19:28:29.163574 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:29.163603 688921 retry.go:31] will retry after 2.118749439s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:29.813858 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml
W0725 19:28:29.882682 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:29.882762 688921 retry.go:31] will retry after 2.509534704s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:30.085229 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
W0725 19:28:30.187228 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:30.187267 688921 retry.go:31] will retry after 3.768348502s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:30.861981 688921 node_ready.go:53] error getting node "old-k8s-version-262689": Get "https://192.168.85.2:8443/api/v1/nodes/old-k8s-version-262689": dial tcp 192.168.85.2:8443: connect: connection refused
I0725 19:28:31.230564 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
I0725 19:28:31.283305 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml
W0725 19:28:31.328879 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:31.328918 688921 retry.go:31] will retry after 2.575529832s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
W0725 19:28:31.413252 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:31.413281 688921 retry.go:31] will retry after 3.929044818s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:32.393288 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml
W0725 19:28:32.505460 688921 addons.go:457] apply failed, will retry: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:32.505496 688921 retry.go:31] will retry after 4.058820604s: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
I0725 19:28:32.862162 688921 node_ready.go:53] error getting node "old-k8s-version-262689": Get "https://192.168.85.2:8443/api/v1/nodes/old-k8s-version-262689": dial tcp 192.168.85.2:8443: connect: connection refused
I0725 19:28:33.905244 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml
I0725 19:28:33.956668 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml
I0725 19:28:35.342865 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml
I0725 19:28:36.565177 688921 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml
I0725 19:28:41.940648 688921 node_ready.go:49] node "old-k8s-version-262689" has status "Ready":"True"
I0725 19:28:41.940672 688921 node_ready.go:38] duration metric: took 17.579930551s for node "old-k8s-version-262689" to be "Ready" ...
I0725 19:28:41.940681 688921 pod_ready.go:35] extra waiting up to 6m0s for all system-critical pods including labels [k8s-app=kube-dns component=etcd component=kube-apiserver component=kube-controller-manager k8s-app=kube-proxy component=kube-scheduler] to be "Ready" ...
I0725 19:28:42.323112 688921 pod_ready.go:78] waiting up to 6m0s for pod "coredns-74ff55c5b-djgf4" in "kube-system" namespace to be "Ready" ...
I0725 19:28:42.758916 688921 pod_ready.go:92] pod "coredns-74ff55c5b-djgf4" in "kube-system" namespace has status "Ready":"True"
I0725 19:28:42.759006 688921 pod_ready.go:81] duration metric: took 435.805641ms for pod "coredns-74ff55c5b-djgf4" in "kube-system" namespace to be "Ready" ...
I0725 19:28:42.759035 688921 pod_ready.go:78] waiting up to 6m0s for pod "etcd-old-k8s-version-262689" in "kube-system" namespace to be "Ready" ...
I0725 19:28:42.961624 688921 pod_ready.go:92] pod "etcd-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"True"
I0725 19:28:42.961697 688921 pod_ready.go:81] duration metric: took 202.639557ms for pod "etcd-old-k8s-version-262689" in "kube-system" namespace to be "Ready" ...
I0725 19:28:42.961725 688921 pod_ready.go:78] waiting up to 6m0s for pod "kube-apiserver-old-k8s-version-262689" in "kube-system" namespace to be "Ready" ...
I0725 19:28:43.118968 688921 pod_ready.go:92] pod "kube-apiserver-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"True"
I0725 19:28:43.119042 688921 pod_ready.go:81] duration metric: took 157.297064ms for pod "kube-apiserver-old-k8s-version-262689" in "kube-system" namespace to be "Ready" ...
I0725 19:28:43.119071 688921 pod_ready.go:78] waiting up to 6m0s for pod "kube-controller-manager-old-k8s-version-262689" in "kube-system" namespace to be "Ready" ...
I0725 19:28:43.166152 688921 pod_ready.go:92] pod "kube-controller-manager-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"True"
I0725 19:28:43.166228 688921 pod_ready.go:81] duration metric: took 47.124737ms for pod "kube-controller-manager-old-k8s-version-262689" in "kube-system" namespace to be "Ready" ...
I0725 19:28:43.166254 688921 pod_ready.go:78] waiting up to 6m0s for pod "kube-proxy-srbcv" in "kube-system" namespace to be "Ready" ...
I0725 19:28:43.222129 688921 pod_ready.go:92] pod "kube-proxy-srbcv" in "kube-system" namespace has status "Ready":"True"
I0725 19:28:43.222193 688921 pod_ready.go:81] duration metric: took 55.917971ms for pod "kube-proxy-srbcv" in "kube-system" namespace to be "Ready" ...
I0725 19:28:43.222228 688921 pod_ready.go:78] waiting up to 6m0s for pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace to be "Ready" ...
I0725 19:28:45.258604 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:28:45.862292 688921 ssh_runner.go:235] Completed: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/metrics-apiservice.yaml -f /etc/kubernetes/addons/metrics-server-deployment.yaml -f /etc/kubernetes/addons/metrics-server-rbac.yaml -f /etc/kubernetes/addons/metrics-server-service.yaml: (11.957008671s)
I0725 19:28:45.862343 688921 addons.go:475] Verifying addon metrics-server=true in "old-k8s-version-262689"
I0725 19:28:46.359586 688921 ssh_runner.go:235] Completed: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/dashboard-ns.yaml -f /etc/kubernetes/addons/dashboard-clusterrole.yaml -f /etc/kubernetes/addons/dashboard-clusterrolebinding.yaml -f /etc/kubernetes/addons/dashboard-configmap.yaml -f /etc/kubernetes/addons/dashboard-dp.yaml -f /etc/kubernetes/addons/dashboard-role.yaml -f /etc/kubernetes/addons/dashboard-rolebinding.yaml -f /etc/kubernetes/addons/dashboard-sa.yaml -f /etc/kubernetes/addons/dashboard-secret.yaml -f /etc/kubernetes/addons/dashboard-svc.yaml: (12.402858335s)
I0725 19:28:46.359846 688921 ssh_runner.go:235] Completed: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storageclass.yaml: (11.016943916s)
I0725 19:28:46.361896 688921 out.go:177] * Some dashboard features require the metrics-server addon. To enable all features please run:
minikube -p old-k8s-version-262689 addons enable metrics-server
I0725 19:28:46.372528 688921 ssh_runner.go:235] Completed: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.20.0/kubectl apply --force -f /etc/kubernetes/addons/storage-provisioner.yaml: (9.807315722s)
I0725 19:28:46.377014 688921 out.go:177] * Enabled addons: metrics-server, dashboard, storage-provisioner, default-storageclass
I0725 19:28:46.378786 688921 addons.go:510] duration metric: took 22.211380884s for enable addons: enabled=[metrics-server dashboard storage-provisioner default-storageclass]
I0725 19:28:47.728957 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:28:50.243655 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:28:52.730611 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:28:55.228972 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:28:57.244538 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:28:59.735814 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:02.026448 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:04.229038 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:06.728468 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:08.729167 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:11.236067 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:13.729844 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:16.228242 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:18.228983 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:20.229832 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:22.252852 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:24.728531 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:26.728849 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:28.729367 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:31.228766 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:33.730049 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:35.730470 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:38.228977 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:40.229059 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:42.250447 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:44.728457 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:46.729106 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:48.730669 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:51.247704 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:53.728691 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:56.229029 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:29:58.729004 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:00.729425 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:03.229772 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:05.729634 688921 pod_ready.go:102] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:06.729150 688921 pod_ready.go:92] pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace has status "Ready":"True"
I0725 19:30:06.729179 688921 pod_ready.go:81] duration metric: took 1m23.506930201s for pod "kube-scheduler-old-k8s-version-262689" in "kube-system" namespace to be "Ready" ...
I0725 19:30:06.729195 688921 pod_ready.go:78] waiting up to 6m0s for pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace to be "Ready" ...
I0725 19:30:08.735776 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:10.736156 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:13.235326 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:15.235416 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:17.236004 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:19.242484 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:21.735211 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:23.735945 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:26.239215 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:28.735197 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:30.735933 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:32.779740 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:35.236193 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:37.735911 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:40.247025 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:42.735548 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:44.736880 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:47.235587 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:49.236663 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:51.735450 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:53.736551 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:56.235526 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:30:58.236093 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:00.306674 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:02.735417 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:04.735811 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:07.236266 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:09.735835 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:11.736351 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:14.236188 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:16.237316 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:18.737138 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:21.236225 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:23.735346 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:26.235191 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:28.239449 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:30.736010 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:33.281074 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:35.736132 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:37.736197 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:40.236564 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:42.237400 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:44.735933 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:47.235692 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:49.236477 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:51.735236 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:54.240405 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:56.736199 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:31:59.235148 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:01.237060 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:03.738340 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:06.236978 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:08.736271 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:10.739517 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:13.245022 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:15.735552 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:18.235721 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:20.236538 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:22.735744 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:25.235722 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:27.236640 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:29.736285 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:32.235366 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:34.236596 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:36.735534 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:38.736334 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:41.235939 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:43.734884 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:45.736417 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:48.235478 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:50.236445 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:52.736899 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:54.737092 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:57.236723 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:32:59.244812 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:01.740019 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:04.236083 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:06.735974 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:09.235848 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:11.235931 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:13.236135 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:15.236538 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:17.237479 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:19.735300 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:21.736670 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:24.269025 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:26.736978 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:29.235104 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:31.235396 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:33.236475 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:35.236629 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:37.237325 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:39.239015 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:41.737833 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:44.238271 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:46.737113 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:49.236415 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:51.736605 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:54.236461 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:56.735944 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:59.235650 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:01.736227 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:03.736798 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:06.235979 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:06.737497 688921 pod_ready.go:81] duration metric: took 4m0.008287476s for pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace to be "Ready" ...
E0725 19:34:06.737526 688921 pod_ready.go:66] WaitExtra: waitPodCondition: context deadline exceeded
I0725 19:34:06.737535 688921 pod_ready.go:38] duration metric: took 5m24.79684369s for extra waiting for all system-critical and pods with labels [k8s-app=kube-dns component=etcd component=kube-apiserver component=kube-controller-manager k8s-app=kube-proxy component=kube-scheduler] to be "Ready" ...
I0725 19:34:06.737548 688921 api_server.go:52] waiting for apiserver process to appear ...
I0725 19:34:06.737576 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0725 19:34:06.737650 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0725 19:34:06.807692 688921 cri.go:89] found id: "8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7"
I0725 19:34:06.807723 688921 cri.go:89] found id: "a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8"
I0725 19:34:06.807729 688921 cri.go:89] found id: ""
I0725 19:34:06.807736 688921 logs.go:276] 2 containers: [8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7 a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8]
I0725 19:34:06.807794 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:06.818733 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:06.822775 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0725 19:34:06.822852 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0725 19:34:06.888510 688921 cri.go:89] found id: "1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967"
I0725 19:34:06.888542 688921 cri.go:89] found id: "bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6"
I0725 19:34:06.888547 688921 cri.go:89] found id: ""
I0725 19:34:06.888557 688921 logs.go:276] 2 containers: [1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967 bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6]
I0725 19:34:06.888612 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:06.892832 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:06.897974 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0725 19:34:06.898094 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0725 19:34:06.959812 688921 cri.go:89] found id: "aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05"
I0725 19:34:06.959838 688921 cri.go:89] found id: "6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8"
I0725 19:34:06.959844 688921 cri.go:89] found id: ""
I0725 19:34:06.959851 688921 logs.go:276] 2 containers: [aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05 6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8]
I0725 19:34:06.959956 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:06.964200 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:06.968411 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0725 19:34:06.968494 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0725 19:34:07.033397 688921 cri.go:89] found id: "fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1"
I0725 19:34:07.033436 688921 cri.go:89] found id: "2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34"
I0725 19:34:07.033442 688921 cri.go:89] found id: ""
I0725 19:34:07.033450 688921 logs.go:276] 2 containers: [fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1 2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34]
I0725 19:34:07.033516 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.043063 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.048413 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0725 19:34:07.048505 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0725 19:34:07.111832 688921 cri.go:89] found id: "09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98"
I0725 19:34:07.111912 688921 cri.go:89] found id: "5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f"
I0725 19:34:07.111931 688921 cri.go:89] found id: ""
I0725 19:34:07.111952 688921 logs.go:276] 2 containers: [09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98 5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f]
I0725 19:34:07.112033 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.117008 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.121337 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0725 19:34:07.121457 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0725 19:34:07.187319 688921 cri.go:89] found id: "83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5"
I0725 19:34:07.187384 688921 cri.go:89] found id: "092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb"
I0725 19:34:07.187401 688921 cri.go:89] found id: ""
I0725 19:34:07.187423 688921 logs.go:276] 2 containers: [83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5 092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb]
I0725 19:34:07.187507 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.192267 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.196794 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kindnet Namespaces:[]}
I0725 19:34:07.196914 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kindnet
I0725 19:34:07.271398 688921 cri.go:89] found id: "ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01"
I0725 19:34:07.271471 688921 cri.go:89] found id: "687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8"
I0725 19:34:07.271489 688921 cri.go:89] found id: ""
I0725 19:34:07.271511 688921 logs.go:276] 2 containers: [ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01 687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8]
I0725 19:34:07.271595 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.275933 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.281359 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0725 19:34:07.281487 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0725 19:34:07.377349 688921 cri.go:89] found id: "1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e"
I0725 19:34:07.377413 688921 cri.go:89] found id: "6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985"
I0725 19:34:07.377432 688921 cri.go:89] found id: ""
I0725 19:34:07.377454 688921 logs.go:276] 2 containers: [1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e 6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985]
I0725 19:34:07.377542 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.383264 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.388959 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0725 19:34:07.389087 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0725 19:34:07.461197 688921 cri.go:89] found id: "22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc"
I0725 19:34:07.461216 688921 cri.go:89] found id: ""
I0725 19:34:07.461223 688921 logs.go:276] 1 containers: [22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc]
I0725 19:34:07.461279 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.465404 688921 logs.go:123] Gathering logs for storage-provisioner [1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e] ...
I0725 19:34:07.465428 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e"
I0725 19:34:07.547164 688921 logs.go:123] Gathering logs for container status ...
I0725 19:34:07.547233 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0725 19:34:07.624426 688921 logs.go:123] Gathering logs for kubelet ...
I0725 19:34:07.624507 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0725 19:34:07.703403 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:42 old-k8s-version-262689 kubelet[657]: E0725 19:28:42.036852 657 reflector.go:138] object-"kube-system"/"kindnet-token-vxqld": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kindnet-token-vxqld" is forbidden: User "system:node:old-k8s-version-262689" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-262689' and this object
W0725 19:34:07.703640 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:42 old-k8s-version-262689 kubelet[657]: E0725 19:28:42.037214 657 reflector.go:138] object-"kube-system"/"kube-proxy-token-2gxrg": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kube-proxy-token-2gxrg" is forbidden: User "system:node:old-k8s-version-262689" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-262689' and this object
W0725 19:34:07.707435 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:45 old-k8s-version-262689 kubelet[657]: E0725 19:28:45.149554 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:07.707634 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:46 old-k8s-version-262689 kubelet[657]: E0725 19:28:46.003537 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.710803 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:57 old-k8s-version-262689 kubelet[657]: E0725 19:28:57.257602 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:07.713278 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:09 old-k8s-version-262689 kubelet[657]: E0725 19:29:09.369957 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.713674 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:10 old-k8s-version-262689 kubelet[657]: E0725 19:29:10.383955 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.713885 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:11 old-k8s-version-262689 kubelet[657]: E0725 19:29:11.215410 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.714611 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:16 old-k8s-version-262689 kubelet[657]: E0725 19:29:16.271158 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.715106 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:17 old-k8s-version-262689 kubelet[657]: E0725 19:29:17.406387 657 pod_workers.go:191] Error syncing pod 192a4c32-53cd-4ce7-ba80-a523469b645d ("storage-provisioner_kube-system(192a4c32-53cd-4ce7-ba80-a523469b645d)"), skipping: failed to "StartContainer" for "storage-provisioner" with CrashLoopBackOff: "back-off 10s restarting failed container=storage-provisioner pod=storage-provisioner_kube-system(192a4c32-53cd-4ce7-ba80-a523469b645d)"
W0725 19:34:07.718311 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:26 old-k8s-version-262689 kubelet[657]: E0725 19:29:26.224127 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:07.719087 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:29 old-k8s-version-262689 kubelet[657]: E0725 19:29:29.440381 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.719629 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:36 old-k8s-version-262689 kubelet[657]: E0725 19:29:36.271666 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.719827 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:37 old-k8s-version-262689 kubelet[657]: E0725 19:29:37.215365 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.720195 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:51 old-k8s-version-262689 kubelet[657]: E0725 19:29:51.215710 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.720693 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:51 old-k8s-version-262689 kubelet[657]: E0725 19:29:51.524343 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.721057 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:56 old-k8s-version-262689 kubelet[657]: E0725 19:29:56.271047 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.721244 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:04 old-k8s-version-262689 kubelet[657]: E0725 19:30:04.215943 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.721572 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:11 old-k8s-version-262689 kubelet[657]: E0725 19:30:11.215709 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.724232 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:19 old-k8s-version-262689 kubelet[657]: E0725 19:30:19.231937 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:07.724588 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:25 old-k8s-version-262689 kubelet[657]: E0725 19:30:25.214643 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.724789 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:32 old-k8s-version-262689 kubelet[657]: E0725 19:30:32.216336 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.725458 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:40 old-k8s-version-262689 kubelet[657]: E0725 19:30:40.656202 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.725659 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:46 old-k8s-version-262689 kubelet[657]: E0725 19:30:46.215221 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.726057 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:46 old-k8s-version-262689 kubelet[657]: E0725 19:30:46.271828 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.726400 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:01 old-k8s-version-262689 kubelet[657]: E0725 19:31:01.215458 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.726633 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:01 old-k8s-version-262689 kubelet[657]: E0725 19:31:01.215611 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.726841 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:13 old-k8s-version-262689 kubelet[657]: E0725 19:31:13.215172 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.727236 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:14 old-k8s-version-262689 kubelet[657]: E0725 19:31:14.214851 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.727622 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:26 old-k8s-version-262689 kubelet[657]: E0725 19:31:26.215358 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.727848 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:27 old-k8s-version-262689 kubelet[657]: E0725 19:31:27.215041 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.728224 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:37 old-k8s-version-262689 kubelet[657]: E0725 19:31:37.215248 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.731097 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:40 old-k8s-version-262689 kubelet[657]: E0725 19:31:40.223791 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:07.731472 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:52 old-k8s-version-262689 kubelet[657]: E0725 19:31:52.217037 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.731662 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:54 old-k8s-version-262689 kubelet[657]: E0725 19:31:54.225403 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.731924 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:05 old-k8s-version-262689 kubelet[657]: E0725 19:32:05.215333 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.732602 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:07 old-k8s-version-262689 kubelet[657]: E0725 19:32:07.888105 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.732942 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:16 old-k8s-version-262689 kubelet[657]: E0725 19:32:16.271463 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.733127 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:19 old-k8s-version-262689 kubelet[657]: E0725 19:32:19.215018 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.733595 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:28 old-k8s-version-262689 kubelet[657]: E0725 19:32:28.214662 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.733787 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:31 old-k8s-version-262689 kubelet[657]: E0725 19:32:31.215144 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.734174 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:41 old-k8s-version-262689 kubelet[657]: E0725 19:32:41.214882 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.734408 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:46 old-k8s-version-262689 kubelet[657]: E0725 19:32:46.215198 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.734844 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:52 old-k8s-version-262689 kubelet[657]: E0725 19:32:52.215208 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.735085 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:59 old-k8s-version-262689 kubelet[657]: E0725 19:32:59.215146 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.735513 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:06 old-k8s-version-262689 kubelet[657]: E0725 19:33:06.214773 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.735733 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:12 old-k8s-version-262689 kubelet[657]: E0725 19:33:12.219911 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.736070 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:17 old-k8s-version-262689 kubelet[657]: E0725 19:33:17.215219 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.736255 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:25 old-k8s-version-262689 kubelet[657]: E0725 19:33:25.215021 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.736581 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:32 old-k8s-version-262689 kubelet[657]: E0725 19:33:32.216211 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.736764 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:38 old-k8s-version-262689 kubelet[657]: E0725 19:33:38.215054 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.737090 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:45 old-k8s-version-262689 kubelet[657]: E0725 19:33:45.216141 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.737377 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:50 old-k8s-version-262689 kubelet[657]: E0725 19:33:50.215161 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.737718 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:58 old-k8s-version-262689 kubelet[657]: E0725 19:33:58.214660 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.737917 688921 logs.go:138] Found kubelet problem: Jul 25 19:34:03 old-k8s-version-262689 kubelet[657]: E0725 19:34:03.215098 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
I0725 19:34:07.737927 688921 logs.go:123] Gathering logs for etcd [1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967] ...
I0725 19:34:07.737943 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967"
I0725 19:34:07.822664 688921 logs.go:123] Gathering logs for kube-scheduler [fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1] ...
I0725 19:34:07.822740 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1"
I0725 19:34:07.879095 688921 logs.go:123] Gathering logs for kindnet [ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01] ...
I0725 19:34:07.879168 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01"
I0725 19:34:07.955648 688921 logs.go:123] Gathering logs for kindnet [687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8] ...
I0725 19:34:07.955730 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8"
I0725 19:34:08.037629 688921 logs.go:123] Gathering logs for kube-apiserver [8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7] ...
I0725 19:34:08.037704 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7"
I0725 19:34:08.114974 688921 logs.go:123] Gathering logs for kube-scheduler [2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34] ...
I0725 19:34:08.115040 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34"
I0725 19:34:08.198695 688921 logs.go:123] Gathering logs for kube-controller-manager [83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5] ...
I0725 19:34:08.198774 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5"
I0725 19:34:08.332924 688921 logs.go:123] Gathering logs for kube-controller-manager [092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb] ...
I0725 19:34:08.333002 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb"
I0725 19:34:08.461781 688921 logs.go:123] Gathering logs for kube-proxy [5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f] ...
I0725 19:34:08.461879 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f"
I0725 19:34:08.540031 688921 logs.go:123] Gathering logs for storage-provisioner [6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985] ...
I0725 19:34:08.540060 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985"
I0725 19:34:08.583359 688921 logs.go:123] Gathering logs for containerd ...
I0725 19:34:08.583437 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0725 19:34:08.659982 688921 logs.go:123] Gathering logs for dmesg ...
I0725 19:34:08.660061 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0725 19:34:08.680183 688921 logs.go:123] Gathering logs for kube-apiserver [a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8] ...
I0725 19:34:08.680216 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8"
I0725 19:34:08.758790 688921 logs.go:123] Gathering logs for coredns [6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8] ...
I0725 19:34:08.758823 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8"
I0725 19:34:08.810366 688921 logs.go:123] Gathering logs for kube-proxy [09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98] ...
I0725 19:34:08.810401 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98"
I0725 19:34:08.855821 688921 logs.go:123] Gathering logs for describe nodes ...
I0725 19:34:08.855850 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.20.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
I0725 19:34:09.049486 688921 logs.go:123] Gathering logs for etcd [bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6] ...
I0725 19:34:09.049516 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6"
I0725 19:34:09.101862 688921 logs.go:123] Gathering logs for coredns [aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05] ...
I0725 19:34:09.101895 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05"
I0725 19:34:09.150649 688921 logs.go:123] Gathering logs for kubernetes-dashboard [22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc] ...
I0725 19:34:09.150683 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc"
I0725 19:34:09.195050 688921 out.go:304] Setting ErrFile to fd 2...
I0725 19:34:09.195080 688921 out.go:338] TERM=,COLORTERM=, which probably does not support color
W0725 19:34:09.195179 688921 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0725 19:34:09.195327 688921 out.go:239] Jul 25 19:33:38 old-k8s-version-262689 kubelet[657]: E0725 19:33:38.215054 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Jul 25 19:33:38 old-k8s-version-262689 kubelet[657]: E0725 19:33:38.215054 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:09.195345 688921 out.go:239] Jul 25 19:33:45 old-k8s-version-262689 kubelet[657]: E0725 19:33:45.216141 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
Jul 25 19:33:45 old-k8s-version-262689 kubelet[657]: E0725 19:33:45.216141 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:09.195424 688921 out.go:239] Jul 25 19:33:50 old-k8s-version-262689 kubelet[657]: E0725 19:33:50.215161 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Jul 25 19:33:50 old-k8s-version-262689 kubelet[657]: E0725 19:33:50.215161 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:09.195440 688921 out.go:239] Jul 25 19:33:58 old-k8s-version-262689 kubelet[657]: E0725 19:33:58.214660 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
Jul 25 19:33:58 old-k8s-version-262689 kubelet[657]: E0725 19:33:58.214660 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:09.195465 688921 out.go:239] Jul 25 19:34:03 old-k8s-version-262689 kubelet[657]: E0725 19:34:03.215098 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Jul 25 19:34:03 old-k8s-version-262689 kubelet[657]: E0725 19:34:03.215098 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
I0725 19:34:09.195473 688921 out.go:304] Setting ErrFile to fd 2...
I0725 19:34:09.195484 688921 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0725 19:34:19.196744 688921 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0725 19:34:19.209410 688921 api_server.go:72] duration metric: took 5m55.042327442s to wait for apiserver process to appear ...
I0725 19:34:19.209435 688921 api_server.go:88] waiting for apiserver healthz status ...
I0725 19:34:19.209472 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0725 19:34:19.209531 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0725 19:34:19.247922 688921 cri.go:89] found id: "8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7"
I0725 19:34:19.247942 688921 cri.go:89] found id: "a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8"
I0725 19:34:19.247947 688921 cri.go:89] found id: ""
I0725 19:34:19.247954 688921 logs.go:276] 2 containers: [8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7 a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8]
I0725 19:34:19.248012 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.252109 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.255778 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0725 19:34:19.255850 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0725 19:34:19.295831 688921 cri.go:89] found id: "1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967"
I0725 19:34:19.295853 688921 cri.go:89] found id: "bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6"
I0725 19:34:19.295859 688921 cri.go:89] found id: ""
I0725 19:34:19.295866 688921 logs.go:276] 2 containers: [1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967 bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6]
I0725 19:34:19.295924 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.300077 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.303811 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0725 19:34:19.303883 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0725 19:34:19.349146 688921 cri.go:89] found id: "aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05"
I0725 19:34:19.349170 688921 cri.go:89] found id: "6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8"
I0725 19:34:19.349176 688921 cri.go:89] found id: ""
I0725 19:34:19.349183 688921 logs.go:276] 2 containers: [aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05 6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8]
I0725 19:34:19.349247 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.353162 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.356919 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0725 19:34:19.357009 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0725 19:34:19.396994 688921 cri.go:89] found id: "fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1"
I0725 19:34:19.397014 688921 cri.go:89] found id: "2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34"
I0725 19:34:19.397018 688921 cri.go:89] found id: ""
I0725 19:34:19.397025 688921 logs.go:276] 2 containers: [fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1 2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34]
I0725 19:34:19.397084 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.401204 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.405026 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0725 19:34:19.405158 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0725 19:34:19.446347 688921 cri.go:89] found id: "09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98"
I0725 19:34:19.446370 688921 cri.go:89] found id: "5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f"
I0725 19:34:19.446375 688921 cri.go:89] found id: ""
I0725 19:34:19.446383 688921 logs.go:276] 2 containers: [09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98 5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f]
I0725 19:34:19.446443 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.450086 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.453624 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0725 19:34:19.453731 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0725 19:34:19.492956 688921 cri.go:89] found id: "83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5"
I0725 19:34:19.492981 688921 cri.go:89] found id: "092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb"
I0725 19:34:19.492985 688921 cri.go:89] found id: ""
I0725 19:34:19.492993 688921 logs.go:276] 2 containers: [83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5 092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb]
I0725 19:34:19.493051 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.497081 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.500953 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kindnet Namespaces:[]}
I0725 19:34:19.501052 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kindnet
I0725 19:34:19.546766 688921 cri.go:89] found id: "ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01"
I0725 19:34:19.546791 688921 cri.go:89] found id: "687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8"
I0725 19:34:19.546797 688921 cri.go:89] found id: ""
I0725 19:34:19.546804 688921 logs.go:276] 2 containers: [ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01 687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8]
I0725 19:34:19.546860 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.550701 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.554230 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0725 19:34:19.554305 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0725 19:34:19.594222 688921 cri.go:89] found id: "1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e"
I0725 19:34:19.594256 688921 cri.go:89] found id: "6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985"
I0725 19:34:19.594261 688921 cri.go:89] found id: ""
I0725 19:34:19.594269 688921 logs.go:276] 2 containers: [1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e 6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985]
I0725 19:34:19.594337 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.598412 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.602080 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0725 19:34:19.602161 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0725 19:34:19.643928 688921 cri.go:89] found id: "22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc"
I0725 19:34:19.643955 688921 cri.go:89] found id: ""
I0725 19:34:19.643963 688921 logs.go:276] 1 containers: [22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc]
I0725 19:34:19.644027 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.647751 688921 logs.go:123] Gathering logs for container status ...
I0725 19:34:19.647778 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0725 19:34:19.690394 688921 logs.go:123] Gathering logs for etcd [1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967] ...
I0725 19:34:19.690510 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967"
I0725 19:34:19.732117 688921 logs.go:123] Gathering logs for kube-proxy [09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98] ...
I0725 19:34:19.732146 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98"
I0725 19:34:19.780472 688921 logs.go:123] Gathering logs for kube-controller-manager [83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5] ...
I0725 19:34:19.780499 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5"
I0725 19:34:19.851807 688921 logs.go:123] Gathering logs for kindnet [687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8] ...
I0725 19:34:19.851890 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8"
I0725 19:34:19.909403 688921 logs.go:123] Gathering logs for storage-provisioner [1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e] ...
I0725 19:34:19.909441 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e"
I0725 19:34:19.947085 688921 logs.go:123] Gathering logs for kubernetes-dashboard [22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc] ...
I0725 19:34:19.947112 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc"
I0725 19:34:19.989416 688921 logs.go:123] Gathering logs for kindnet [ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01] ...
I0725 19:34:19.989441 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01"
I0725 19:34:20.080244 688921 logs.go:123] Gathering logs for containerd ...
I0725 19:34:20.080284 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0725 19:34:20.148363 688921 logs.go:123] Gathering logs for kubelet ...
I0725 19:34:20.148399 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0725 19:34:20.209971 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:42 old-k8s-version-262689 kubelet[657]: E0725 19:28:42.036852 657 reflector.go:138] object-"kube-system"/"kindnet-token-vxqld": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kindnet-token-vxqld" is forbidden: User "system:node:old-k8s-version-262689" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-262689' and this object
W0725 19:34:20.210213 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:42 old-k8s-version-262689 kubelet[657]: E0725 19:28:42.037214 657 reflector.go:138] object-"kube-system"/"kube-proxy-token-2gxrg": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kube-proxy-token-2gxrg" is forbidden: User "system:node:old-k8s-version-262689" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-262689' and this object
W0725 19:34:20.213793 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:45 old-k8s-version-262689 kubelet[657]: E0725 19:28:45.149554 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:20.215174 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:46 old-k8s-version-262689 kubelet[657]: E0725 19:28:46.003537 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.218024 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:57 old-k8s-version-262689 kubelet[657]: E0725 19:28:57.257602 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:20.220231 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:09 old-k8s-version-262689 kubelet[657]: E0725 19:29:09.369957 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.220930 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:10 old-k8s-version-262689 kubelet[657]: E0725 19:29:10.383955 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.221254 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:11 old-k8s-version-262689 kubelet[657]: E0725 19:29:11.215410 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.222117 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:16 old-k8s-version-262689 kubelet[657]: E0725 19:29:16.271158 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.222720 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:17 old-k8s-version-262689 kubelet[657]: E0725 19:29:17.406387 657 pod_workers.go:191] Error syncing pod 192a4c32-53cd-4ce7-ba80-a523469b645d ("storage-provisioner_kube-system(192a4c32-53cd-4ce7-ba80-a523469b645d)"), skipping: failed to "StartContainer" for "storage-provisioner" with CrashLoopBackOff: "back-off 10s restarting failed container=storage-provisioner pod=storage-provisioner_kube-system(192a4c32-53cd-4ce7-ba80-a523469b645d)"
W0725 19:34:20.226773 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:26 old-k8s-version-262689 kubelet[657]: E0725 19:29:26.224127 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:20.227465 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:29 old-k8s-version-262689 kubelet[657]: E0725 19:29:29.440381 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.227957 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:36 old-k8s-version-262689 kubelet[657]: E0725 19:29:36.271666 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.228162 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:37 old-k8s-version-262689 kubelet[657]: E0725 19:29:37.215365 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.228549 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:51 old-k8s-version-262689 kubelet[657]: E0725 19:29:51.215710 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.229042 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:51 old-k8s-version-262689 kubelet[657]: E0725 19:29:51.524343 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.229493 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:56 old-k8s-version-262689 kubelet[657]: E0725 19:29:56.271047 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.229718 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:04 old-k8s-version-262689 kubelet[657]: E0725 19:30:04.215943 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.230164 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:11 old-k8s-version-262689 kubelet[657]: E0725 19:30:11.215709 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.232845 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:19 old-k8s-version-262689 kubelet[657]: E0725 19:30:19.231937 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:20.233247 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:25 old-k8s-version-262689 kubelet[657]: E0725 19:30:25.214643 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.233529 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:32 old-k8s-version-262689 kubelet[657]: E0725 19:30:32.216336 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.234139 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:40 old-k8s-version-262689 kubelet[657]: E0725 19:30:40.656202 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.234363 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:46 old-k8s-version-262689 kubelet[657]: E0725 19:30:46.215221 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.234728 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:46 old-k8s-version-262689 kubelet[657]: E0725 19:30:46.271828 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.235078 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:01 old-k8s-version-262689 kubelet[657]: E0725 19:31:01.215458 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.235295 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:01 old-k8s-version-262689 kubelet[657]: E0725 19:31:01.215611 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.235532 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:13 old-k8s-version-262689 kubelet[657]: E0725 19:31:13.215172 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.235906 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:14 old-k8s-version-262689 kubelet[657]: E0725 19:31:14.214851 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.236251 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:26 old-k8s-version-262689 kubelet[657]: E0725 19:31:26.215358 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.236454 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:27 old-k8s-version-262689 kubelet[657]: E0725 19:31:27.215041 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.236868 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:37 old-k8s-version-262689 kubelet[657]: E0725 19:31:37.215248 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.240631 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:40 old-k8s-version-262689 kubelet[657]: E0725 19:31:40.223791 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:20.241195 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:52 old-k8s-version-262689 kubelet[657]: E0725 19:31:52.217037 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.241446 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:54 old-k8s-version-262689 kubelet[657]: E0725 19:31:54.225403 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.241652 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:05 old-k8s-version-262689 kubelet[657]: E0725 19:32:05.215333 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.242309 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:07 old-k8s-version-262689 kubelet[657]: E0725 19:32:07.888105 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.242659 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:16 old-k8s-version-262689 kubelet[657]: E0725 19:32:16.271463 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.242899 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:19 old-k8s-version-262689 kubelet[657]: E0725 19:32:19.215018 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.243284 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:28 old-k8s-version-262689 kubelet[657]: E0725 19:32:28.214662 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.243635 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:31 old-k8s-version-262689 kubelet[657]: E0725 19:32:31.215144 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.244269 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:41 old-k8s-version-262689 kubelet[657]: E0725 19:32:41.214882 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.244545 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:46 old-k8s-version-262689 kubelet[657]: E0725 19:32:46.215198 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.244932 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:52 old-k8s-version-262689 kubelet[657]: E0725 19:32:52.215208 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.245138 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:59 old-k8s-version-262689 kubelet[657]: E0725 19:32:59.215146 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.245498 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:06 old-k8s-version-262689 kubelet[657]: E0725 19:33:06.214773 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.245703 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:12 old-k8s-version-262689 kubelet[657]: E0725 19:33:12.219911 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.246101 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:17 old-k8s-version-262689 kubelet[657]: E0725 19:33:17.215219 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.246327 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:25 old-k8s-version-262689 kubelet[657]: E0725 19:33:25.215021 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.246766 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:32 old-k8s-version-262689 kubelet[657]: E0725 19:33:32.216211 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.247012 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:38 old-k8s-version-262689 kubelet[657]: E0725 19:33:38.215054 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.247382 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:45 old-k8s-version-262689 kubelet[657]: E0725 19:33:45.216141 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.247585 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:50 old-k8s-version-262689 kubelet[657]: E0725 19:33:50.215161 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.247936 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:58 old-k8s-version-262689 kubelet[657]: E0725 19:33:58.214660 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.248172 688921 logs.go:138] Found kubelet problem: Jul 25 19:34:03 old-k8s-version-262689 kubelet[657]: E0725 19:34:03.215098 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.248568 688921 logs.go:138] Found kubelet problem: Jul 25 19:34:11 old-k8s-version-262689 kubelet[657]: E0725 19:34:11.214739 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.248773 688921 logs.go:138] Found kubelet problem: Jul 25 19:34:14 old-k8s-version-262689 kubelet[657]: E0725 19:34:14.215104 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
I0725 19:34:20.248804 688921 logs.go:123] Gathering logs for describe nodes ...
I0725 19:34:20.248838 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.20.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
I0725 19:34:20.394413 688921 logs.go:123] Gathering logs for kube-apiserver [8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7] ...
I0725 19:34:20.394445 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7"
I0725 19:34:20.454291 688921 logs.go:123] Gathering logs for kube-apiserver [a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8] ...
I0725 19:34:20.454325 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8"
I0725 19:34:20.507516 688921 logs.go:123] Gathering logs for coredns [6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8] ...
I0725 19:34:20.507595 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8"
I0725 19:34:20.549222 688921 logs.go:123] Gathering logs for kube-scheduler [fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1] ...
I0725 19:34:20.549250 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1"
I0725 19:34:20.600314 688921 logs.go:123] Gathering logs for etcd [bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6] ...
I0725 19:34:20.600352 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6"
I0725 19:34:20.657965 688921 logs.go:123] Gathering logs for kube-proxy [5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f] ...
I0725 19:34:20.657998 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f"
I0725 19:34:20.702882 688921 logs.go:123] Gathering logs for kube-controller-manager [092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb] ...
I0725 19:34:20.702913 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb"
I0725 19:34:20.766370 688921 logs.go:123] Gathering logs for dmesg ...
I0725 19:34:20.766446 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0725 19:34:20.794437 688921 logs.go:123] Gathering logs for coredns [aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05] ...
I0725 19:34:20.794537 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05"
I0725 19:34:20.838887 688921 logs.go:123] Gathering logs for kube-scheduler [2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34] ...
I0725 19:34:20.839007 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34"
I0725 19:34:20.898415 688921 logs.go:123] Gathering logs for storage-provisioner [6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985] ...
I0725 19:34:20.898447 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985"
I0725 19:34:20.947486 688921 out.go:304] Setting ErrFile to fd 2...
I0725 19:34:20.947512 688921 out.go:338] TERM=,COLORTERM=, which probably does not support color
W0725 19:34:20.947573 688921 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0725 19:34:20.947582 688921 out.go:239] Jul 25 19:33:50 old-k8s-version-262689 kubelet[657]: E0725 19:33:50.215161 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Jul 25 19:33:50 old-k8s-version-262689 kubelet[657]: E0725 19:33:50.215161 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.947592 688921 out.go:239] Jul 25 19:33:58 old-k8s-version-262689 kubelet[657]: E0725 19:33:58.214660 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
Jul 25 19:33:58 old-k8s-version-262689 kubelet[657]: E0725 19:33:58.214660 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.947606 688921 out.go:239] Jul 25 19:34:03 old-k8s-version-262689 kubelet[657]: E0725 19:34:03.215098 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Jul 25 19:34:03 old-k8s-version-262689 kubelet[657]: E0725 19:34:03.215098 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.947612 688921 out.go:239] Jul 25 19:34:11 old-k8s-version-262689 kubelet[657]: E0725 19:34:11.214739 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
Jul 25 19:34:11 old-k8s-version-262689 kubelet[657]: E0725 19:34:11.214739 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.947623 688921 out.go:239] Jul 25 19:34:14 old-k8s-version-262689 kubelet[657]: E0725 19:34:14.215104 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Jul 25 19:34:14 old-k8s-version-262689 kubelet[657]: E0725 19:34:14.215104 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
I0725 19:34:20.947630 688921 out.go:304] Setting ErrFile to fd 2...
I0725 19:34:20.947635 688921 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0725 19:34:30.948693 688921 api_server.go:253] Checking apiserver healthz at https://192.168.85.2:8443/healthz ...
I0725 19:34:30.959121 688921 api_server.go:279] https://192.168.85.2:8443/healthz returned 200:
ok
I0725 19:34:30.961497 688921 out.go:177]
W0725 19:34:30.963511 688921 out.go:239] X Exiting due to K8S_UNHEALTHY_CONTROL_PLANE: wait 6m0s for node: wait for healthy API server: controlPlane never updated to v1.20.0
X Exiting due to K8S_UNHEALTHY_CONTROL_PLANE: wait 6m0s for node: wait for healthy API server: controlPlane never updated to v1.20.0
W0725 19:34:30.963612 688921 out.go:239] * Suggestion: Control Plane could not update, try minikube delete --all --purge
* Suggestion: Control Plane could not update, try minikube delete --all --purge
W0725 19:34:30.963654 688921 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/11417
* Related issue: https://github.com/kubernetes/minikube/issues/11417
W0725 19:34:30.963661 688921 out.go:239] *
*
W0725 19:34:30.964606 688921 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0725 19:34:30.966662 688921 out.go:177]
** /stderr **
start_stop_delete_test.go:259: failed to start minikube post-stop. args "out/minikube-linux-arm64 start -p old-k8s-version-262689 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --container-runtime=containerd --kubernetes-version=v1.20.0": exit status 102
helpers_test.go:222: -----------------------post-mortem--------------------------------
helpers_test.go:230: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/SecondStart]: docker inspect <======
helpers_test.go:231: (dbg) Run: docker inspect old-k8s-version-262689
helpers_test.go:235: (dbg) docker inspect old-k8s-version-262689:
-- stdout --
[
{
"Id": "ad3e5b171b2995bead1aa162fead29fa218841c43c7ac6e27fd72de330c3b7b7",
"Created": "2024-07-25T19:25:36.611162491Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 689127,
"ExitCode": 0,
"Error": "",
"StartedAt": "2024-07-25T19:28:17.34778438Z",
"FinishedAt": "2024-07-25T19:28:16.254703516Z"
},
"Image": "sha256:2cd84ab2172023a68162f38a55db46353562cea41552fd8e8bdec97b31b2c495",
"ResolvConfPath": "/var/lib/docker/containers/ad3e5b171b2995bead1aa162fead29fa218841c43c7ac6e27fd72de330c3b7b7/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/ad3e5b171b2995bead1aa162fead29fa218841c43c7ac6e27fd72de330c3b7b7/hostname",
"HostsPath": "/var/lib/docker/containers/ad3e5b171b2995bead1aa162fead29fa218841c43c7ac6e27fd72de330c3b7b7/hosts",
"LogPath": "/var/lib/docker/containers/ad3e5b171b2995bead1aa162fead29fa218841c43c7ac6e27fd72de330c3b7b7/ad3e5b171b2995bead1aa162fead29fa218841c43c7ac6e27fd72de330c3b7b7-json.log",
"Name": "/old-k8s-version-262689",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "unconfined",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro",
"old-k8s-version-262689:/var"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "old-k8s-version-262689",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "host",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 2306867200,
"NanoCpus": 2000000000,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": [],
"BlkioDeviceWriteBps": [],
"BlkioDeviceReadIOps": [],
"BlkioDeviceWriteIOps": [],
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 4613734400,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": [],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/418c0882e2e9198d8e5e6de7f15076ddcd713d0702378867f7f17ee3d3e4bd14-init/diff:/var/lib/docker/overlay2/2f35ea3391cd80b943121d1a194672f5d1b43fa71caefe855446e579999be65e/diff",
"MergedDir": "/var/lib/docker/overlay2/418c0882e2e9198d8e5e6de7f15076ddcd713d0702378867f7f17ee3d3e4bd14/merged",
"UpperDir": "/var/lib/docker/overlay2/418c0882e2e9198d8e5e6de7f15076ddcd713d0702378867f7f17ee3d3e4bd14/diff",
"WorkDir": "/var/lib/docker/overlay2/418c0882e2e9198d8e5e6de7f15076ddcd713d0702378867f7f17ee3d3e4bd14/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "volume",
"Name": "old-k8s-version-262689",
"Source": "/var/lib/docker/volumes/old-k8s-version-262689/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
},
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "old-k8s-version-262689",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"22/tcp": {},
"2376/tcp": {},
"32443/tcp": {},
"5000/tcp": {},
"8443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "old-k8s-version-262689",
"name.minikube.sigs.k8s.io": "old-k8s-version-262689",
"role.minikube.sigs.k8s.io": ""
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "d7e1c6dc23f4ac0332976b6b8cb4b01481c4a77bc34a0164b3903a33b560f775",
"SandboxKey": "/var/run/docker/netns/d7e1c6dc23f4",
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33496"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33497"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33500"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33498"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33499"
}
]
},
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"old-k8s-version-262689": {
"IPAMConfig": {
"IPv4Address": "192.168.85.2"
},
"Links": null,
"Aliases": null,
"MacAddress": "02:42:c0:a8:55:02",
"DriverOpts": null,
"NetworkID": "54e9c52dec53efbe8fef6070e5abd7d8dc7bb9d79eeee2dd4b697ef53bd44da4",
"EndpointID": "eee26b7868c125af8799690bb326cd4a1231d171ed3292e9e222cff3ed6121c9",
"Gateway": "192.168.85.1",
"IPAddress": "192.168.85.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"DNSNames": [
"old-k8s-version-262689",
"ad3e5b171b29"
]
}
}
}
}
]
-- /stdout --
helpers_test.go:239: (dbg) Run: out/minikube-linux-arm64 status --format={{.Host}} -p old-k8s-version-262689 -n old-k8s-version-262689
helpers_test.go:244: <<< TestStartStop/group/old-k8s-version/serial/SecondStart FAILED: start of post-mortem logs <<<
helpers_test.go:245: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/SecondStart]: minikube logs <======
helpers_test.go:247: (dbg) Run: out/minikube-linux-arm64 -p old-k8s-version-262689 logs -n 25
helpers_test.go:247: (dbg) Done: out/minikube-linux-arm64 -p old-k8s-version-262689 logs -n 25: (2.253920501s)
helpers_test.go:252: TestStartStop/group/old-k8s-version/serial/SecondStart logs:
-- stdout --
==> Audit <==
|---------|--------------------------------------------------------|------------------------|---------|---------|---------------------|---------------------|
| Command | Args | Profile | User | Version | Start Time | End Time |
|---------|--------------------------------------------------------|------------------------|---------|---------|---------------------|---------------------|
| ssh | -p bridge-212266 sudo | bridge-212266 | jenkins | v1.33.1 | 25 Jul 24 19:26 UTC | 25 Jul 24 19:26 UTC |
| | systemctl status containerd | | | | | |
| | --all --full --no-pager | | | | | |
| ssh | -p bridge-212266 sudo | bridge-212266 | jenkins | v1.33.1 | 25 Jul 24 19:26 UTC | 25 Jul 24 19:26 UTC |
| | systemctl cat containerd | | | | | |
| | --no-pager | | | | | |
| ssh | -p bridge-212266 sudo cat | bridge-212266 | jenkins | v1.33.1 | 25 Jul 24 19:26 UTC | 25 Jul 24 19:26 UTC |
| | /lib/systemd/system/containerd.service | | | | | |
| ssh | -p bridge-212266 sudo cat | bridge-212266 | jenkins | v1.33.1 | 25 Jul 24 19:26 UTC | 25 Jul 24 19:26 UTC |
| | /etc/containerd/config.toml | | | | | |
| ssh | -p bridge-212266 sudo | bridge-212266 | jenkins | v1.33.1 | 25 Jul 24 19:26 UTC | 25 Jul 24 19:26 UTC |
| | containerd config dump | | | | | |
| ssh | -p bridge-212266 sudo | bridge-212266 | jenkins | v1.33.1 | 25 Jul 24 19:26 UTC | |
| | systemctl status crio --all | | | | | |
| | --full --no-pager | | | | | |
| ssh | -p bridge-212266 sudo | bridge-212266 | jenkins | v1.33.1 | 25 Jul 24 19:26 UTC | 25 Jul 24 19:26 UTC |
| | systemctl cat crio --no-pager | | | | | |
| ssh | -p bridge-212266 sudo find | bridge-212266 | jenkins | v1.33.1 | 25 Jul 24 19:26 UTC | 25 Jul 24 19:26 UTC |
| | /etc/crio -type f -exec sh -c | | | | | |
| | 'echo {}; cat {}' \; | | | | | |
| ssh | -p bridge-212266 sudo crio | bridge-212266 | jenkins | v1.33.1 | 25 Jul 24 19:26 UTC | 25 Jul 24 19:26 UTC |
| | config | | | | | |
| delete | -p bridge-212266 | bridge-212266 | jenkins | v1.33.1 | 25 Jul 24 19:26 UTC | 25 Jul 24 19:26 UTC |
| start | -p no-preload-143817 --memory=2200 | no-preload-143817 | jenkins | v1.33.1 | 25 Jul 24 19:26 UTC | 25 Jul 24 19:28 UTC |
| | --alsologtostderr --wait=true | | | | | |
| | --preload=false --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| | --kubernetes-version=v1.31.0-beta.0 | | | | | |
| addons | enable metrics-server -p old-k8s-version-262689 | old-k8s-version-262689 | jenkins | v1.33.1 | 25 Jul 24 19:28 UTC | 25 Jul 24 19:28 UTC |
| | --images=MetricsServer=registry.k8s.io/echoserver:1.4 | | | | | |
| | --registries=MetricsServer=fake.domain | | | | | |
| stop | -p old-k8s-version-262689 | old-k8s-version-262689 | jenkins | v1.33.1 | 25 Jul 24 19:28 UTC | 25 Jul 24 19:28 UTC |
| | --alsologtostderr -v=3 | | | | | |
| addons | enable dashboard -p old-k8s-version-262689 | old-k8s-version-262689 | jenkins | v1.33.1 | 25 Jul 24 19:28 UTC | 25 Jul 24 19:28 UTC |
| | --images=MetricsScraper=registry.k8s.io/echoserver:1.4 | | | | | |
| start | -p old-k8s-version-262689 | old-k8s-version-262689 | jenkins | v1.33.1 | 25 Jul 24 19:28 UTC | |
| | --memory=2200 | | | | | |
| | --alsologtostderr --wait=true | | | | | |
| | --kvm-network=default | | | | | |
| | --kvm-qemu-uri=qemu:///system | | | | | |
| | --disable-driver-mounts | | | | | |
| | --keep-context=false | | | | | |
| | --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| | --kubernetes-version=v1.20.0 | | | | | |
| addons | enable metrics-server -p no-preload-143817 | no-preload-143817 | jenkins | v1.33.1 | 25 Jul 24 19:28 UTC | 25 Jul 24 19:28 UTC |
| | --images=MetricsServer=registry.k8s.io/echoserver:1.4 | | | | | |
| | --registries=MetricsServer=fake.domain | | | | | |
| stop | -p no-preload-143817 | no-preload-143817 | jenkins | v1.33.1 | 25 Jul 24 19:28 UTC | 25 Jul 24 19:28 UTC |
| | --alsologtostderr -v=3 | | | | | |
| addons | enable dashboard -p no-preload-143817 | no-preload-143817 | jenkins | v1.33.1 | 25 Jul 24 19:28 UTC | 25 Jul 24 19:28 UTC |
| | --images=MetricsScraper=registry.k8s.io/echoserver:1.4 | | | | | |
| start | -p no-preload-143817 --memory=2200 | no-preload-143817 | jenkins | v1.33.1 | 25 Jul 24 19:28 UTC | 25 Jul 24 19:33 UTC |
| | --alsologtostderr --wait=true | | | | | |
| | --preload=false --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| | --kubernetes-version=v1.31.0-beta.0 | | | | | |
| image | no-preload-143817 image list | no-preload-143817 | jenkins | v1.33.1 | 25 Jul 24 19:33 UTC | 25 Jul 24 19:33 UTC |
| | --format=json | | | | | |
| pause | -p no-preload-143817 | no-preload-143817 | jenkins | v1.33.1 | 25 Jul 24 19:33 UTC | 25 Jul 24 19:33 UTC |
| | --alsologtostderr -v=1 | | | | | |
| unpause | -p no-preload-143817 | no-preload-143817 | jenkins | v1.33.1 | 25 Jul 24 19:33 UTC | 25 Jul 24 19:33 UTC |
| | --alsologtostderr -v=1 | | | | | |
| delete | -p no-preload-143817 | no-preload-143817 | jenkins | v1.33.1 | 25 Jul 24 19:33 UTC | 25 Jul 24 19:33 UTC |
| delete | -p no-preload-143817 | no-preload-143817 | jenkins | v1.33.1 | 25 Jul 24 19:33 UTC | 25 Jul 24 19:33 UTC |
| start | -p embed-certs-240166 | embed-certs-240166 | jenkins | v1.33.1 | 25 Jul 24 19:33 UTC | 25 Jul 24 19:34 UTC |
| | --memory=2200 | | | | | |
| | --alsologtostderr --wait=true | | | | | |
| | --embed-certs --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| | --kubernetes-version=v1.30.3 | | | | | |
|---------|--------------------------------------------------------|------------------------|---------|---------|---------------------|---------------------|
==> Last Start <==
Log file created at: 2024/07/25 19:33:21
Running on machine: ip-172-31-30-239
Binary: Built with gc go1.22.5 for linux/arm64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0725 19:33:21.286087 698181 out.go:291] Setting OutFile to fd 1 ...
I0725 19:33:21.286271 698181 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0725 19:33:21.286285 698181 out.go:304] Setting ErrFile to fd 2...
I0725 19:33:21.286292 698181 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0725 19:33:21.286571 698181 root.go:338] Updating PATH: /home/jenkins/minikube-integration/19326-431487/.minikube/bin
I0725 19:33:21.287054 698181 out.go:298] Setting JSON to false
I0725 19:33:21.288186 698181 start.go:129] hostinfo: {"hostname":"ip-172-31-30-239","uptime":11750,"bootTime":1721924251,"procs":234,"os":"linux","platform":"ubuntu","platformFamily":"debian","platformVersion":"20.04","kernelVersion":"5.15.0-1065-aws","kernelArch":"aarch64","virtualizationSystem":"","virtualizationRole":"","hostId":"92f46a7d-c249-4c12-924a-77f64874c910"}
I0725 19:33:21.288258 698181 start.go:139] virtualization:
I0725 19:33:21.290661 698181 out.go:177] * [embed-certs-240166] minikube v1.33.1 on Ubuntu 20.04 (arm64)
I0725 19:33:21.292892 698181 out.go:177] - MINIKUBE_LOCATION=19326
I0725 19:33:21.293014 698181 notify.go:220] Checking for updates...
I0725 19:33:21.296003 698181 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0725 19:33:21.297804 698181 out.go:177] - KUBECONFIG=/home/jenkins/minikube-integration/19326-431487/kubeconfig
I0725 19:33:21.299665 698181 out.go:177] - MINIKUBE_HOME=/home/jenkins/minikube-integration/19326-431487/.minikube
I0725 19:33:21.301320 698181 out.go:177] - MINIKUBE_BIN=out/minikube-linux-arm64
I0725 19:33:21.303031 698181 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0725 19:33:21.305404 698181 config.go:182] Loaded profile config "old-k8s-version-262689": Driver=docker, ContainerRuntime=containerd, KubernetesVersion=v1.20.0
I0725 19:33:21.305965 698181 driver.go:392] Setting default libvirt URI to qemu:///system
I0725 19:33:21.326604 698181 docker.go:123] docker version: linux-27.1.1:Docker Engine - Community
I0725 19:33:21.326835 698181 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0725 19:33:21.392667 698181 info.go:266] docker info: {ID:6ZPO:QZND:VNGE:LUKL:4Y3K:XELL:AAX4:2GTK:E6LM:MPRN:3ZXR:TTMR Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:35 OomKillDisable:true NGoroutines:53 SystemTime:2024-07-25 19:33:21.378968035 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1065-aws OperatingSystem:Ubuntu 20.04.6 LTS OSType:linux Architecture:aar
ch64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:2 MemTotal:8214900736 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ip-172-31-30-239 Labels:[] ExperimentalBuild:false ServerVersion:27.1.1 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41 Expected:2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41} RuncCommit:{ID:v1.1.13-0-g58aa920 Expected:v1.1.13-0-g58aa920} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=builtin] ProductLicense: Warnings:<nil> ServerErro
rs:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.16.1] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.29.1]] Warnings:<nil>}}
I0725 19:33:21.392783 698181 docker.go:307] overlay module found
I0725 19:33:21.394788 698181 out.go:177] * Using the docker driver based on user configuration
I0725 19:33:21.396340 698181 start.go:297] selected driver: docker
I0725 19:33:21.396357 698181 start.go:901] validating driver "docker" against <nil>
I0725 19:33:21.396372 698181 start.go:912] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0725 19:33:21.397068 698181 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0725 19:33:21.449340 698181 info.go:266] docker info: {ID:6ZPO:QZND:VNGE:LUKL:4Y3K:XELL:AAX4:2GTK:E6LM:MPRN:3ZXR:TTMR Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:34 OomKillDisable:true NGoroutines:53 SystemTime:2024-07-25 19:33:21.439211779 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1065-aws OperatingSystem:Ubuntu 20.04.6 LTS OSType:linux Architecture:aar
ch64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:2 MemTotal:8214900736 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ip-172-31-30-239 Labels:[] ExperimentalBuild:false ServerVersion:27.1.1 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41 Expected:2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41} RuncCommit:{ID:v1.1.13-0-g58aa920 Expected:v1.1.13-0-g58aa920} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=builtin] ProductLicense: Warnings:<nil> ServerErro
rs:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.16.1] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.29.1]] Warnings:<nil>}}
I0725 19:33:21.449520 698181 start_flags.go:310] no existing cluster config was found, will generate one from the flags
I0725 19:33:21.449756 698181 start_flags.go:947] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0725 19:33:21.451573 698181 out.go:177] * Using Docker driver with root privileges
I0725 19:33:21.453299 698181 cni.go:84] Creating CNI manager for ""
I0725 19:33:21.453318 698181 cni.go:143] "docker" driver + "containerd" runtime found, recommending kindnet
I0725 19:33:21.453330 698181 start_flags.go:319] Found "CNI" CNI - setting NetworkPlugin=cni
I0725 19:33:21.453440 698181 start.go:340] cluster config:
{Name:embed-certs-240166 KeepContext:false EmbedCerts:true MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.30.3 ClusterName:embed-certs-240166 Namespace:default APIServerHAVIP: APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local Contain
erRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.30.3 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: Stati
cIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0725 19:33:21.455461 698181 out.go:177] * Starting "embed-certs-240166" primary control-plane node in "embed-certs-240166" cluster
I0725 19:33:21.457046 698181 cache.go:121] Beginning downloading kic base image for docker with containerd
I0725 19:33:21.458776 698181 out.go:177] * Pulling base image v0.0.44-1721902582-19326 ...
I0725 19:33:21.460583 698181 preload.go:131] Checking if preload exists for k8s version v1.30.3 and runtime containerd
I0725 19:33:21.460645 698181 preload.go:146] Found local preload: /home/jenkins/minikube-integration/19326-431487/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.3-containerd-overlay2-arm64.tar.lz4
I0725 19:33:21.460666 698181 cache.go:56] Caching tarball of preloaded images
I0725 19:33:21.460678 698181 image.go:79] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 in local docker daemon
I0725 19:33:21.460746 698181 preload.go:172] Found /home/jenkins/minikube-integration/19326-431487/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.3-containerd-overlay2-arm64.tar.lz4 in cache, skipping download
I0725 19:33:21.460756 698181 cache.go:59] Finished verifying existence of preloaded tar for v1.30.3 on containerd
I0725 19:33:21.460855 698181 profile.go:143] Saving config to /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/config.json ...
I0725 19:33:21.460905 698181 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/config.json: {Name:mka6ae37130518f5717245e655531644dd928580 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
W0725 19:33:21.480030 698181 image.go:95] image gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 is of wrong architecture
I0725 19:33:21.480052 698181 cache.go:149] Downloading gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 to local cache
I0725 19:33:21.480139 698181 image.go:63] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 in local cache directory
I0725 19:33:21.480163 698181 image.go:66] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 in local cache directory, skipping pull
I0725 19:33:21.480173 698181 image.go:135] gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 exists in cache, skipping pull
I0725 19:33:21.480181 698181 cache.go:152] successfully saved gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 as a tarball
I0725 19:33:21.480190 698181 cache.go:162] Loading gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 from local cache
I0725 19:33:21.603025 698181 cache.go:164] successfully loaded and using gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 from cached tarball
I0725 19:33:21.603068 698181 cache.go:194] Successfully downloaded all kic artifacts
I0725 19:33:21.603104 698181 start.go:360] acquireMachinesLock for embed-certs-240166: {Name:mk12ce5baa5d6e39306cbb300f74fcabccebd6da Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0725 19:33:21.604201 698181 start.go:364] duration metric: took 1.070474ms to acquireMachinesLock for "embed-certs-240166"
I0725 19:33:21.604247 698181 start.go:93] Provisioning new machine with config: &{Name:embed-certs-240166 KeepContext:false EmbedCerts:true MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.30.3 ClusterName:embed-certs-240166 Namespace:default APIServerHAVIP: APIServe
rName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.30.3 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableM
etrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s} &{Name: IP: Port:8443 KubernetesVersion:v1.30.3 ContainerRuntime:containerd ControlPlane:true Worker:true}
I0725 19:33:21.604341 698181 start.go:125] createHost starting for "" (driver="docker")
I0725 19:33:17.237479 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:19.735300 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:21.736670 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:21.606517 698181 out.go:204] * Creating docker container (CPUs=2, Memory=2200MB) ...
I0725 19:33:21.606772 698181 start.go:159] libmachine.API.Create for "embed-certs-240166" (driver="docker")
I0725 19:33:21.606824 698181 client.go:168] LocalClient.Create starting
I0725 19:33:21.606890 698181 main.go:141] libmachine: Reading certificate data from /home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca.pem
I0725 19:33:21.606927 698181 main.go:141] libmachine: Decoding PEM data...
I0725 19:33:21.606996 698181 main.go:141] libmachine: Parsing certificate...
I0725 19:33:21.607068 698181 main.go:141] libmachine: Reading certificate data from /home/jenkins/minikube-integration/19326-431487/.minikube/certs/cert.pem
I0725 19:33:21.607095 698181 main.go:141] libmachine: Decoding PEM data...
I0725 19:33:21.607113 698181 main.go:141] libmachine: Parsing certificate...
I0725 19:33:21.607496 698181 cli_runner.go:164] Run: docker network inspect embed-certs-240166 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
W0725 19:33:21.623331 698181 cli_runner.go:211] docker network inspect embed-certs-240166 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1
I0725 19:33:21.623413 698181 network_create.go:284] running [docker network inspect embed-certs-240166] to gather additional debugging logs...
I0725 19:33:21.623431 698181 cli_runner.go:164] Run: docker network inspect embed-certs-240166
W0725 19:33:21.638402 698181 cli_runner.go:211] docker network inspect embed-certs-240166 returned with exit code 1
I0725 19:33:21.638438 698181 network_create.go:287] error running [docker network inspect embed-certs-240166]: docker network inspect embed-certs-240166: exit status 1
stdout:
[]
stderr:
Error response from daemon: network embed-certs-240166 not found
I0725 19:33:21.638453 698181 network_create.go:289] output of [docker network inspect embed-certs-240166]: -- stdout --
[]
-- /stdout --
** stderr **
Error response from daemon: network embed-certs-240166 not found
** /stderr **
I0725 19:33:21.638569 698181 cli_runner.go:164] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0725 19:33:21.661771 698181 network.go:211] skipping subnet 192.168.49.0/24 that is taken: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName:br-a821fe35c4f7 IfaceIPv4:192.168.49.1 IfaceMTU:1500 IfaceMAC:02:42:35:60:6d:c3} reservation:<nil>}
I0725 19:33:21.662251 698181 network.go:211] skipping subnet 192.168.58.0/24 that is taken: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 IsPrivate:true Interface:{IfaceName:br-f7c6a289108c IfaceIPv4:192.168.58.1 IfaceMTU:1500 IfaceMAC:02:42:5a:53:02:1b} reservation:<nil>}
I0725 19:33:21.662628 698181 network.go:211] skipping subnet 192.168.67.0/24 that is taken: &{IP:192.168.67.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.67.0/24 Gateway:192.168.67.1 ClientMin:192.168.67.2 ClientMax:192.168.67.254 Broadcast:192.168.67.255 IsPrivate:true Interface:{IfaceName:br-8b48f7ba5d2a IfaceIPv4:192.168.67.1 IfaceMTU:1500 IfaceMAC:02:42:4c:9c:01:cd} reservation:<nil>}
I0725 19:33:21.663216 698181 network.go:206] using free private subnet 192.168.76.0/24: &{IP:192.168.76.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.76.0/24 Gateway:192.168.76.1 ClientMin:192.168.76.2 ClientMax:192.168.76.254 Broadcast:192.168.76.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0x4001842fd0}
I0725 19:33:21.663246 698181 network_create.go:124] attempt to create docker network embed-certs-240166 192.168.76.0/24 with gateway 192.168.76.1 and MTU of 1500 ...
I0725 19:33:21.663305 698181 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.76.0/24 --gateway=192.168.76.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=1500 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=embed-certs-240166 embed-certs-240166
I0725 19:33:21.757746 698181 network_create.go:108] docker network embed-certs-240166 192.168.76.0/24 created
I0725 19:33:21.757783 698181 kic.go:121] calculated static IP "192.168.76.2" for the "embed-certs-240166" container
I0725 19:33:21.757858 698181 cli_runner.go:164] Run: docker ps -a --format {{.Names}}
I0725 19:33:21.787127 698181 cli_runner.go:164] Run: docker volume create embed-certs-240166 --label name.minikube.sigs.k8s.io=embed-certs-240166 --label created_by.minikube.sigs.k8s.io=true
I0725 19:33:21.803931 698181 oci.go:103] Successfully created a docker volume embed-certs-240166
I0725 19:33:21.804026 698181 cli_runner.go:164] Run: docker run --rm --name embed-certs-240166-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=embed-certs-240166 --entrypoint /usr/bin/test -v embed-certs-240166:/var gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 -d /var/lib
I0725 19:33:22.468647 698181 oci.go:107] Successfully prepared a docker volume embed-certs-240166
I0725 19:33:22.468701 698181 preload.go:131] Checking if preload exists for k8s version v1.30.3 and runtime containerd
I0725 19:33:22.468722 698181 kic.go:194] Starting extracting preloaded images to volume ...
I0725 19:33:22.468804 698181 cli_runner.go:164] Run: docker run --rm --entrypoint /usr/bin/tar -v /home/jenkins/minikube-integration/19326-431487/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.3-containerd-overlay2-arm64.tar.lz4:/preloaded.tar:ro -v embed-certs-240166:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 -I lz4 -xf /preloaded.tar -C /extractDir
I0725 19:33:24.269025 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:26.736978 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:27.306852 698181 cli_runner.go:217] Completed: docker run --rm --entrypoint /usr/bin/tar -v /home/jenkins/minikube-integration/19326-431487/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.3-containerd-overlay2-arm64.tar.lz4:/preloaded.tar:ro -v embed-certs-240166:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 -I lz4 -xf /preloaded.tar -C /extractDir: (4.837993165s)
I0725 19:33:27.306884 698181 kic.go:203] duration metric: took 4.838157797s to extract preloaded images to volume ...
W0725 19:33:27.307078 698181 cgroups_linux.go:77] Your kernel does not support swap limit capabilities or the cgroup is not mounted.
I0725 19:33:27.307199 698181 cli_runner.go:164] Run: docker info --format "'{{json .SecurityOptions}}'"
I0725 19:33:27.360658 698181 cli_runner.go:164] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname embed-certs-240166 --name embed-certs-240166 --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=embed-certs-240166 --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=embed-certs-240166 --network embed-certs-240166 --ip 192.168.76.2 --volume embed-certs-240166:/var --security-opt apparmor=unconfined --memory=2200mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7
I0725 19:33:27.689655 698181 cli_runner.go:164] Run: docker container inspect embed-certs-240166 --format={{.State.Running}}
I0725 19:33:27.718081 698181 cli_runner.go:164] Run: docker container inspect embed-certs-240166 --format={{.State.Status}}
I0725 19:33:27.745269 698181 cli_runner.go:164] Run: docker exec embed-certs-240166 stat /var/lib/dpkg/alternatives/iptables
I0725 19:33:27.804169 698181 oci.go:144] the created container "embed-certs-240166" has a running status.
I0725 19:33:27.804195 698181 kic.go:225] Creating ssh key for kic: /home/jenkins/minikube-integration/19326-431487/.minikube/machines/embed-certs-240166/id_rsa...
I0725 19:33:28.366584 698181 kic_runner.go:191] docker (temp): /home/jenkins/minikube-integration/19326-431487/.minikube/machines/embed-certs-240166/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0725 19:33:28.398525 698181 cli_runner.go:164] Run: docker container inspect embed-certs-240166 --format={{.State.Status}}
I0725 19:33:28.428736 698181 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0725 19:33:28.428768 698181 kic_runner.go:114] Args: [docker exec --privileged embed-certs-240166 chown docker:docker /home/docker/.ssh/authorized_keys]
I0725 19:33:28.502372 698181 cli_runner.go:164] Run: docker container inspect embed-certs-240166 --format={{.State.Status}}
I0725 19:33:28.539007 698181 machine.go:94] provisionDockerMachine start ...
I0725 19:33:28.539107 698181 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-240166
I0725 19:33:28.564018 698181 main.go:141] libmachine: Using SSH client type: native
I0725 19:33:28.564290 698181 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3e2cd0] 0x3e5530 <nil> [] 0s} 127.0.0.1 33506 <nil> <nil>}
I0725 19:33:28.564348 698181 main.go:141] libmachine: About to run SSH command:
hostname
I0725 19:33:28.740176 698181 main.go:141] libmachine: SSH cmd err, output: <nil>: embed-certs-240166
I0725 19:33:28.740206 698181 ubuntu.go:169] provisioning hostname "embed-certs-240166"
I0725 19:33:28.740572 698181 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-240166
I0725 19:33:28.780462 698181 main.go:141] libmachine: Using SSH client type: native
I0725 19:33:28.780712 698181 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3e2cd0] 0x3e5530 <nil> [] 0s} 127.0.0.1 33506 <nil> <nil>}
I0725 19:33:28.780731 698181 main.go:141] libmachine: About to run SSH command:
sudo hostname embed-certs-240166 && echo "embed-certs-240166" | sudo tee /etc/hostname
I0725 19:33:28.949018 698181 main.go:141] libmachine: SSH cmd err, output: <nil>: embed-certs-240166
I0725 19:33:28.949159 698181 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-240166
I0725 19:33:28.968513 698181 main.go:141] libmachine: Using SSH client type: native
I0725 19:33:28.968767 698181 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3e2cd0] 0x3e5530 <nil> [] 0s} 127.0.0.1 33506 <nil> <nil>}
I0725 19:33:28.968789 698181 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sembed-certs-240166' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 embed-certs-240166/g' /etc/hosts;
else
echo '127.0.1.1 embed-certs-240166' | sudo tee -a /etc/hosts;
fi
fi
I0725 19:33:29.110865 698181 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0725 19:33:29.110894 698181 ubuntu.go:175] set auth options {CertDir:/home/jenkins/minikube-integration/19326-431487/.minikube CaCertPath:/home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca.pem CaPrivateKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/server.pem ServerKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/server-key.pem ClientKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/home/jenkins/minikube-integration/19326-431487/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/home/jenkins/minikube-integration/19326-431487/.minikube}
I0725 19:33:29.111028 698181 ubuntu.go:177] setting up certificates
I0725 19:33:29.111043 698181 provision.go:84] configureAuth start
I0725 19:33:29.111142 698181 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" embed-certs-240166
I0725 19:33:29.143811 698181 provision.go:143] copyHostCerts
I0725 19:33:29.143877 698181 exec_runner.go:144] found /home/jenkins/minikube-integration/19326-431487/.minikube/ca.pem, removing ...
I0725 19:33:29.143891 698181 exec_runner.go:203] rm: /home/jenkins/minikube-integration/19326-431487/.minikube/ca.pem
I0725 19:33:29.143964 698181 exec_runner.go:151] cp: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca.pem --> /home/jenkins/minikube-integration/19326-431487/.minikube/ca.pem (1082 bytes)
I0725 19:33:29.144060 698181 exec_runner.go:144] found /home/jenkins/minikube-integration/19326-431487/.minikube/cert.pem, removing ...
I0725 19:33:29.144070 698181 exec_runner.go:203] rm: /home/jenkins/minikube-integration/19326-431487/.minikube/cert.pem
I0725 19:33:29.144095 698181 exec_runner.go:151] cp: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/cert.pem --> /home/jenkins/minikube-integration/19326-431487/.minikube/cert.pem (1123 bytes)
I0725 19:33:29.144150 698181 exec_runner.go:144] found /home/jenkins/minikube-integration/19326-431487/.minikube/key.pem, removing ...
I0725 19:33:29.144159 698181 exec_runner.go:203] rm: /home/jenkins/minikube-integration/19326-431487/.minikube/key.pem
I0725 19:33:29.144183 698181 exec_runner.go:151] cp: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/key.pem --> /home/jenkins/minikube-integration/19326-431487/.minikube/key.pem (1679 bytes)
I0725 19:33:29.144245 698181 provision.go:117] generating server cert: /home/jenkins/minikube-integration/19326-431487/.minikube/machines/server.pem ca-key=/home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca.pem private-key=/home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca-key.pem org=jenkins.embed-certs-240166 san=[127.0.0.1 192.168.76.2 embed-certs-240166 localhost minikube]
I0725 19:33:29.506793 698181 provision.go:177] copyRemoteCerts
I0725 19:33:29.506888 698181 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0725 19:33:29.506952 698181 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-240166
I0725 19:33:29.523875 698181 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33506 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/embed-certs-240166/id_rsa Username:docker}
I0725 19:33:29.622986 698181 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1082 bytes)
I0725 19:33:29.651091 698181 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/machines/server.pem --> /etc/docker/server.pem (1220 bytes)
I0725 19:33:29.677554 698181 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1679 bytes)
I0725 19:33:29.702991 698181 provision.go:87] duration metric: took 591.93031ms to configureAuth
I0725 19:33:29.703016 698181 ubuntu.go:193] setting minikube options for container-runtime
I0725 19:33:29.703195 698181 config.go:182] Loaded profile config "embed-certs-240166": Driver=docker, ContainerRuntime=containerd, KubernetesVersion=v1.30.3
I0725 19:33:29.703248 698181 machine.go:97] duration metric: took 1.164200278s to provisionDockerMachine
I0725 19:33:29.703261 698181 client.go:171] duration metric: took 8.096428246s to LocalClient.Create
I0725 19:33:29.703276 698181 start.go:167] duration metric: took 8.096508835s to libmachine.API.Create "embed-certs-240166"
I0725 19:33:29.703283 698181 start.go:293] postStartSetup for "embed-certs-240166" (driver="docker")
I0725 19:33:29.703292 698181 start.go:322] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0725 19:33:29.703343 698181 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0725 19:33:29.703382 698181 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-240166
I0725 19:33:29.720635 698181 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33506 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/embed-certs-240166/id_rsa Username:docker}
I0725 19:33:29.817601 698181 ssh_runner.go:195] Run: cat /etc/os-release
I0725 19:33:29.821712 698181 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0725 19:33:29.821748 698181 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0725 19:33:29.821759 698181 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0725 19:33:29.821766 698181 info.go:137] Remote host: Ubuntu 22.04.4 LTS
I0725 19:33:29.821776 698181 filesync.go:126] Scanning /home/jenkins/minikube-integration/19326-431487/.minikube/addons for local assets ...
I0725 19:33:29.821838 698181 filesync.go:126] Scanning /home/jenkins/minikube-integration/19326-431487/.minikube/files for local assets ...
I0725 19:33:29.821927 698181 filesync.go:149] local asset: /home/jenkins/minikube-integration/19326-431487/.minikube/files/etc/ssl/certs/4368932.pem -> 4368932.pem in /etc/ssl/certs
I0725 19:33:29.822035 698181 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0725 19:33:29.831050 698181 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/files/etc/ssl/certs/4368932.pem --> /etc/ssl/certs/4368932.pem (1708 bytes)
I0725 19:33:29.856799 698181 start.go:296] duration metric: took 153.501464ms for postStartSetup
I0725 19:33:29.857188 698181 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" embed-certs-240166
I0725 19:33:29.874370 698181 profile.go:143] Saving config to /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/config.json ...
I0725 19:33:29.874833 698181 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0725 19:33:29.874888 698181 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-240166
I0725 19:33:29.891612 698181 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33506 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/embed-certs-240166/id_rsa Username:docker}
I0725 19:33:29.987959 698181 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0725 19:33:29.993193 698181 start.go:128] duration metric: took 8.388834764s to createHost
I0725 19:33:29.993215 698181 start.go:83] releasing machines lock for "embed-certs-240166", held for 8.388994629s
I0725 19:33:29.993292 698181 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" embed-certs-240166
I0725 19:33:30.041150 698181 ssh_runner.go:195] Run: cat /version.json
I0725 19:33:30.041216 698181 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-240166
I0725 19:33:30.041498 698181 ssh_runner.go:195] Run: curl -sS -m 2 https://registry.k8s.io/
I0725 19:33:30.041566 698181 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-240166
I0725 19:33:30.066990 698181 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33506 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/embed-certs-240166/id_rsa Username:docker}
I0725 19:33:30.074094 698181 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33506 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/embed-certs-240166/id_rsa Username:docker}
I0725 19:33:30.171464 698181 ssh_runner.go:195] Run: systemctl --version
I0725 19:33:30.310923 698181 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
I0725 19:33:30.315424 698181 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *loopback.conf* -not -name *.mk_disabled -exec sh -c "grep -q loopback {} && ( grep -q name {} || sudo sed -i '/"type": "loopback"/i \ \ \ \ "name": "loopback",' {} ) && sudo sed -i 's|"cniVersion": ".*"|"cniVersion": "1.0.0"|g' {}" ;
I0725 19:33:30.343063 698181 cni.go:230] loopback cni configuration patched: "/etc/cni/net.d/*loopback.conf*" found
I0725 19:33:30.343257 698181 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f ( ( -name *bridge* -or -name *podman* ) -and -not -name *.mk_disabled ) -printf "%!p(MISSING), " -exec sh -c "sudo mv {} {}.mk_disabled" ;
I0725 19:33:30.373709 698181 cni.go:262] disabled [/etc/cni/net.d/87-podman-bridge.conflist, /etc/cni/net.d/100-crio-bridge.conf] bridge cni config(s)
I0725 19:33:30.373735 698181 start.go:495] detecting cgroup driver to use...
I0725 19:33:30.373768 698181 detect.go:187] detected "cgroupfs" cgroup driver on host os
I0725 19:33:30.373827 698181 ssh_runner.go:195] Run: sudo systemctl stop -f crio
I0725 19:33:30.388124 698181 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0725 19:33:30.400545 698181 docker.go:217] disabling cri-docker service (if available) ...
I0725 19:33:30.400611 698181 ssh_runner.go:195] Run: sudo systemctl stop -f cri-docker.socket
I0725 19:33:30.414450 698181 ssh_runner.go:195] Run: sudo systemctl stop -f cri-docker.service
I0725 19:33:30.429800 698181 ssh_runner.go:195] Run: sudo systemctl disable cri-docker.socket
I0725 19:33:30.515245 698181 ssh_runner.go:195] Run: sudo systemctl mask cri-docker.service
I0725 19:33:30.612730 698181 docker.go:233] disabling docker service ...
I0725 19:33:30.612804 698181 ssh_runner.go:195] Run: sudo systemctl stop -f docker.socket
I0725 19:33:30.636014 698181 ssh_runner.go:195] Run: sudo systemctl stop -f docker.service
I0725 19:33:30.648280 698181 ssh_runner.go:195] Run: sudo systemctl disable docker.socket
I0725 19:33:30.743686 698181 ssh_runner.go:195] Run: sudo systemctl mask docker.service
I0725 19:33:30.840320 698181 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service docker
I0725 19:33:30.853028 698181 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %!s(MISSING) "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0725 19:33:30.870703 698181 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.9"|' /etc/containerd/config.toml"
I0725 19:33:30.881325 698181 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I0725 19:33:30.892077 698181 containerd.go:146] configuring containerd to use "cgroupfs" as cgroup driver...
I0725 19:33:30.892167 698181 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = false|g' /etc/containerd/config.toml"
I0725 19:33:30.902689 698181 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0725 19:33:30.912883 698181 ssh_runner.go:195] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I0725 19:33:30.923509 698181 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0725 19:33:30.933999 698181 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0725 19:33:30.943971 698181 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I0725 19:33:30.953964 698181 ssh_runner.go:195] Run: sh -c "sudo sed -i '/^ *enable_unprivileged_ports = .*/d' /etc/containerd/config.toml"
I0725 19:33:30.964381 698181 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)\[plugins."io.containerd.grpc.v1.cri"\]|&\n\1 enable_unprivileged_ports = true|' /etc/containerd/config.toml"
I0725 19:33:30.976608 698181 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0725 19:33:30.988735 698181 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0725 19:33:30.998487 698181 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0725 19:33:31.093478 698181 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0725 19:33:31.245256 698181 start.go:542] Will wait 60s for socket path /run/containerd/containerd.sock
I0725 19:33:31.245359 698181 ssh_runner.go:195] Run: stat /run/containerd/containerd.sock
I0725 19:33:31.249151 698181 start.go:563] Will wait 60s for crictl version
I0725 19:33:31.249245 698181 ssh_runner.go:195] Run: which crictl
I0725 19:33:31.253138 698181 ssh_runner.go:195] Run: sudo /usr/bin/crictl version
I0725 19:33:31.301013 698181 start.go:579] Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: 1.7.19
RuntimeApiVersion: v1
I0725 19:33:31.301145 698181 ssh_runner.go:195] Run: containerd --version
I0725 19:33:31.325398 698181 ssh_runner.go:195] Run: containerd --version
I0725 19:33:31.352075 698181 out.go:177] * Preparing Kubernetes v1.30.3 on containerd 1.7.19 ...
I0725 19:33:29.235104 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:31.235396 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:31.353780 698181 cli_runner.go:164] Run: docker network inspect embed-certs-240166 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0725 19:33:31.370568 698181 ssh_runner.go:195] Run: grep 192.168.76.1 host.minikube.internal$ /etc/hosts
I0725 19:33:31.374335 698181 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.76.1 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0725 19:33:31.386395 698181 kubeadm.go:883] updating cluster {Name:embed-certs-240166 KeepContext:false EmbedCerts:true MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.30.3 ClusterName:embed-certs-240166 Namespace:default APIServerHAVIP: APIServerName:minikubeCA AP
IServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.30.3 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:
false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s} ...
I0725 19:33:31.386522 698181 preload.go:131] Checking if preload exists for k8s version v1.30.3 and runtime containerd
I0725 19:33:31.386587 698181 ssh_runner.go:195] Run: sudo crictl images --output json
I0725 19:33:31.429169 698181 containerd.go:627] all images are preloaded for containerd runtime.
I0725 19:33:31.429194 698181 containerd.go:534] Images already preloaded, skipping extraction
I0725 19:33:31.429258 698181 ssh_runner.go:195] Run: sudo crictl images --output json
I0725 19:33:31.468583 698181 containerd.go:627] all images are preloaded for containerd runtime.
I0725 19:33:31.468607 698181 cache_images.go:84] Images are preloaded, skipping loading
I0725 19:33:31.468616 698181 kubeadm.go:934] updating node { 192.168.76.2 8443 v1.30.3 containerd true true} ...
I0725 19:33:31.468762 698181 kubeadm.go:946] kubelet [Unit]
Wants=containerd.service
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.30.3/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --hostname-override=embed-certs-240166 --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.76.2
[Install]
config:
{KubernetesVersion:v1.30.3 ClusterName:embed-certs-240166 Namespace:default APIServerHAVIP: APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:}
I0725 19:33:31.468840 698181 ssh_runner.go:195] Run: sudo crictl info
I0725 19:33:31.508103 698181 cni.go:84] Creating CNI manager for ""
I0725 19:33:31.508130 698181 cni.go:143] "docker" driver + "containerd" runtime found, recommending kindnet
I0725 19:33:31.508141 698181 kubeadm.go:84] Using pod CIDR: 10.244.0.0/16
I0725 19:33:31.508200 698181 kubeadm.go:181] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.76.2 APIServerPort:8443 KubernetesVersion:v1.30.3 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:embed-certs-240166 NodeName:embed-certs-240166 DNSDomain:cluster.local CRISocket:/run/containerd/containerd.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.76.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.76.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticPo
dPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[containerRuntimeEndpoint:unix:///run/containerd/containerd.sock hairpinMode:hairpin-veth runtimeRequestTimeout:15m] PrependCriSocketUnix:true}
I0725 19:33:31.508369 698181 kubeadm.go:187] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.76.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: unix:///run/containerd/containerd.sock
name: "embed-certs-240166"
kubeletExtraArgs:
node-ip: 192.168.76.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.76.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: mk
controlPlaneEndpoint: control-plane.minikube.internal:8443
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
proxy-refresh-interval: "70000"
kubernetesVersion: v1.30.3
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
containerRuntimeEndpoint: unix:///run/containerd/containerd.sock
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%!"(MISSING)
nodefs.inodesFree: "0%!"(MISSING)
imagefs.available: "0%!"(MISSING)
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0725 19:33:31.508444 698181 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.30.3
I0725 19:33:31.517801 698181 binaries.go:44] Found k8s binaries, skipping transfer
I0725 19:33:31.517878 698181 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0725 19:33:31.527377 698181 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (322 bytes)
I0725 19:33:31.547255 698181 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0725 19:33:31.568491 698181 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2172 bytes)
I0725 19:33:31.588345 698181 ssh_runner.go:195] Run: grep 192.168.76.2 control-plane.minikube.internal$ /etc/hosts
I0725 19:33:31.591995 698181 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.76.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0725 19:33:31.603812 698181 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0725 19:33:31.689689 698181 ssh_runner.go:195] Run: sudo systemctl start kubelet
I0725 19:33:31.704202 698181 certs.go:68] Setting up /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166 for IP: 192.168.76.2
I0725 19:33:31.704226 698181 certs.go:194] generating shared ca certs ...
I0725 19:33:31.704244 698181 certs.go:226] acquiring lock for ca certs: {Name:mk41d7b1e7cb52699a093c81e00768f54d73ad8a Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0725 19:33:31.704417 698181 certs.go:235] skipping valid "minikubeCA" ca cert: /home/jenkins/minikube-integration/19326-431487/.minikube/ca.key
I0725 19:33:31.704485 698181 certs.go:235] skipping valid "proxyClientCA" ca cert: /home/jenkins/minikube-integration/19326-431487/.minikube/proxy-client-ca.key
I0725 19:33:31.704507 698181 certs.go:256] generating profile certs ...
I0725 19:33:31.704581 698181 certs.go:363] generating signed profile cert for "minikube-user": /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/client.key
I0725 19:33:31.704599 698181 crypto.go:68] Generating cert /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/client.crt with IP's: []
I0725 19:33:32.186631 698181 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/client.crt ...
I0725 19:33:32.186661 698181 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/client.crt: {Name:mk9f8e7c4ff5a207e674f3df448020f54f39cd2e Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0725 19:33:32.187468 698181 crypto.go:164] Writing key to /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/client.key ...
I0725 19:33:32.187492 698181 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/client.key: {Name:mk3d9b077ab20a79d3672d18bcd04c1fd79de06d Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0725 19:33:32.188238 698181 certs.go:363] generating signed profile cert for "minikube": /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/apiserver.key.db275045
I0725 19:33:32.188264 698181 crypto.go:68] Generating cert /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/apiserver.crt.db275045 with IP's: [10.96.0.1 127.0.0.1 10.0.0.1 192.168.76.2]
I0725 19:33:33.162876 698181 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/apiserver.crt.db275045 ...
I0725 19:33:33.162909 698181 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/apiserver.crt.db275045: {Name:mkca91b3261b2785acdfcb08b17303cf0cb4bd25 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0725 19:33:33.163685 698181 crypto.go:164] Writing key to /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/apiserver.key.db275045 ...
I0725 19:33:33.163704 698181 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/apiserver.key.db275045: {Name:mk666de744508f02eb76a0e47b56c4ca8edd2188 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0725 19:33:33.163808 698181 certs.go:381] copying /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/apiserver.crt.db275045 -> /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/apiserver.crt
I0725 19:33:33.163893 698181 certs.go:385] copying /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/apiserver.key.db275045 -> /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/apiserver.key
I0725 19:33:33.163962 698181 certs.go:363] generating signed profile cert for "aggregator": /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/proxy-client.key
I0725 19:33:33.163981 698181 crypto.go:68] Generating cert /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/proxy-client.crt with IP's: []
I0725 19:33:33.398078 698181 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/proxy-client.crt ...
I0725 19:33:33.398112 698181 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/proxy-client.crt: {Name:mk5bc0fcba79e4269e2201872e15f982db0851bf Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0725 19:33:33.398323 698181 crypto.go:164] Writing key to /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/proxy-client.key ...
I0725 19:33:33.398341 698181 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/proxy-client.key: {Name:mk61f4809552f61ef52620008e2c1e388c8c971a Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0725 19:33:33.398536 698181 certs.go:484] found cert: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/436893.pem (1338 bytes)
W0725 19:33:33.398584 698181 certs.go:480] ignoring /home/jenkins/minikube-integration/19326-431487/.minikube/certs/436893_empty.pem, impossibly tiny 0 bytes
I0725 19:33:33.398605 698181 certs.go:484] found cert: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca-key.pem (1675 bytes)
I0725 19:33:33.398633 698181 certs.go:484] found cert: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/ca.pem (1082 bytes)
I0725 19:33:33.398662 698181 certs.go:484] found cert: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/cert.pem (1123 bytes)
I0725 19:33:33.398690 698181 certs.go:484] found cert: /home/jenkins/minikube-integration/19326-431487/.minikube/certs/key.pem (1679 bytes)
I0725 19:33:33.398738 698181 certs.go:484] found cert: /home/jenkins/minikube-integration/19326-431487/.minikube/files/etc/ssl/certs/4368932.pem (1708 bytes)
I0725 19:33:33.399446 698181 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0725 19:33:33.424390 698181 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1679 bytes)
I0725 19:33:33.449128 698181 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0725 19:33:33.477115 698181 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1675 bytes)
I0725 19:33:33.503421 698181 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1428 bytes)
I0725 19:33:33.530144 698181 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1675 bytes)
I0725 19:33:33.556098 698181 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0725 19:33:33.583146 698181 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/profiles/embed-certs-240166/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1675 bytes)
I0725 19:33:33.610478 698181 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/files/etc/ssl/certs/4368932.pem --> /usr/share/ca-certificates/4368932.pem (1708 bytes)
I0725 19:33:33.637066 698181 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0725 19:33:33.662610 698181 ssh_runner.go:362] scp /home/jenkins/minikube-integration/19326-431487/.minikube/certs/436893.pem --> /usr/share/ca-certificates/436893.pem (1338 bytes)
I0725 19:33:33.689664 698181 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0725 19:33:33.709484 698181 ssh_runner.go:195] Run: openssl version
I0725 19:33:33.715426 698181 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/4368932.pem && ln -fs /usr/share/ca-certificates/4368932.pem /etc/ssl/certs/4368932.pem"
I0725 19:33:33.725379 698181 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/4368932.pem
I0725 19:33:33.729205 698181 certs.go:528] hashing: -rw-r--r-- 1 root root 1708 Jul 25 18:40 /usr/share/ca-certificates/4368932.pem
I0725 19:33:33.729311 698181 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/4368932.pem
I0725 19:33:33.738249 698181 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/4368932.pem /etc/ssl/certs/3ec20f2e.0"
I0725 19:33:33.748340 698181 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0725 19:33:33.758338 698181 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0725 19:33:33.761876 698181 certs.go:528] hashing: -rw-r--r-- 1 root root 1111 Jul 25 18:30 /usr/share/ca-certificates/minikubeCA.pem
I0725 19:33:33.761971 698181 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0725 19:33:33.769152 698181 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0725 19:33:33.786534 698181 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/436893.pem && ln -fs /usr/share/ca-certificates/436893.pem /etc/ssl/certs/436893.pem"
I0725 19:33:33.798695 698181 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/436893.pem
I0725 19:33:33.802587 698181 certs.go:528] hashing: -rw-r--r-- 1 root root 1338 Jul 25 18:40 /usr/share/ca-certificates/436893.pem
I0725 19:33:33.802701 698181 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/436893.pem
I0725 19:33:33.810466 698181 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/436893.pem /etc/ssl/certs/51391683.0"
I0725 19:33:33.824559 698181 ssh_runner.go:195] Run: stat /var/lib/minikube/certs/apiserver-kubelet-client.crt
I0725 19:33:33.828557 698181 certs.go:399] 'apiserver-kubelet-client' cert doesn't exist, likely first start: stat /var/lib/minikube/certs/apiserver-kubelet-client.crt: Process exited with status 1
stdout:
stderr:
stat: cannot statx '/var/lib/minikube/certs/apiserver-kubelet-client.crt': No such file or directory
I0725 19:33:33.828625 698181 kubeadm.go:392] StartCluster: {Name:embed-certs-240166 KeepContext:false EmbedCerts:true MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.44-1721902582-19326@sha256:540fb5dc7f38be17ff5276a38dfe6c8a4b1d9ba1c27c62244e6eebd7e37696e7 Memory:2200 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.30.3 ClusterName:embed-certs-240166 Namespace:default APIServerHAVIP: APIServerName:minikubeCA APISe
rverNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.30.3 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:fal
se CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0725 19:33:33.828713 698181 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:paused Name: Namespaces:[kube-system]}
I0725 19:33:33.828774 698181 ssh_runner.go:195] Run: sudo -s eval "crictl ps -a --quiet --label io.kubernetes.pod.namespace=kube-system"
I0725 19:33:33.870692 698181 cri.go:89] found id: ""
I0725 19:33:33.870785 698181 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0725 19:33:33.880440 698181 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0725 19:33:33.890210 698181 kubeadm.go:214] ignoring SystemVerification for kubeadm because of docker driver
I0725 19:33:33.890274 698181 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0725 19:33:33.900158 698181 kubeadm.go:155] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0725 19:33:33.900178 698181 kubeadm.go:157] found existing configuration files:
I0725 19:33:33.900234 698181 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/admin.conf
I0725 19:33:33.910104 698181 kubeadm.go:163] "https://control-plane.minikube.internal:8443" may not be in /etc/kubernetes/admin.conf - will remove: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/admin.conf: Process exited with status 2
stdout:
stderr:
grep: /etc/kubernetes/admin.conf: No such file or directory
I0725 19:33:33.910233 698181 ssh_runner.go:195] Run: sudo rm -f /etc/kubernetes/admin.conf
I0725 19:33:33.919375 698181 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/kubelet.conf
I0725 19:33:33.928955 698181 kubeadm.go:163] "https://control-plane.minikube.internal:8443" may not be in /etc/kubernetes/kubelet.conf - will remove: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/kubelet.conf: Process exited with status 2
stdout:
stderr:
grep: /etc/kubernetes/kubelet.conf: No such file or directory
I0725 19:33:33.929052 698181 ssh_runner.go:195] Run: sudo rm -f /etc/kubernetes/kubelet.conf
I0725 19:33:33.937818 698181 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/controller-manager.conf
I0725 19:33:33.947153 698181 kubeadm.go:163] "https://control-plane.minikube.internal:8443" may not be in /etc/kubernetes/controller-manager.conf - will remove: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/controller-manager.conf: Process exited with status 2
stdout:
stderr:
grep: /etc/kubernetes/controller-manager.conf: No such file or directory
I0725 19:33:33.947268 698181 ssh_runner.go:195] Run: sudo rm -f /etc/kubernetes/controller-manager.conf
I0725 19:33:33.956017 698181 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/scheduler.conf
I0725 19:33:33.965557 698181 kubeadm.go:163] "https://control-plane.minikube.internal:8443" may not be in /etc/kubernetes/scheduler.conf - will remove: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
grep: /etc/kubernetes/scheduler.conf: No such file or directory
I0725 19:33:33.965644 698181 ssh_runner.go:195] Run: sudo rm -f /etc/kubernetes/scheduler.conf
I0725 19:33:33.974526 698181 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.30.3:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0725 19:33:34.028995 698181 kubeadm.go:310] [init] Using Kubernetes version: v1.30.3
I0725 19:33:34.029254 698181 kubeadm.go:310] [preflight] Running pre-flight checks
I0725 19:33:34.082611 698181 kubeadm.go:310] [preflight] The system verification failed. Printing the output from the verification:
I0725 19:33:34.082762 698181 kubeadm.go:310] [0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1065-aws[0m
I0725 19:33:34.082827 698181 kubeadm.go:310] [0;37mOS[0m: [0;32mLinux[0m
I0725 19:33:34.082897 698181 kubeadm.go:310] [0;37mCGROUPS_CPU[0m: [0;32menabled[0m
I0725 19:33:34.083025 698181 kubeadm.go:310] [0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
I0725 19:33:34.083108 698181 kubeadm.go:310] [0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
I0725 19:33:34.083187 698181 kubeadm.go:310] [0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
I0725 19:33:34.083265 698181 kubeadm.go:310] [0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
I0725 19:33:34.083353 698181 kubeadm.go:310] [0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
I0725 19:33:34.083425 698181 kubeadm.go:310] [0;37mCGROUPS_PIDS[0m: [0;32menabled[0m
I0725 19:33:34.083505 698181 kubeadm.go:310] [0;37mCGROUPS_HUGETLB[0m: [0;32menabled[0m
I0725 19:33:34.083582 698181 kubeadm.go:310] [0;37mCGROUPS_BLKIO[0m: [0;32menabled[0m
I0725 19:33:34.203513 698181 kubeadm.go:310] [preflight] Pulling images required for setting up a Kubernetes cluster
I0725 19:33:34.203667 698181 kubeadm.go:310] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0725 19:33:34.203790 698181 kubeadm.go:310] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0725 19:33:34.563358 698181 kubeadm.go:310] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0725 19:33:34.566714 698181 out.go:204] - Generating certificates and keys ...
I0725 19:33:34.566905 698181 kubeadm.go:310] [certs] Using existing ca certificate authority
I0725 19:33:34.567068 698181 kubeadm.go:310] [certs] Using existing apiserver certificate and key on disk
I0725 19:33:35.047348 698181 kubeadm.go:310] [certs] Generating "apiserver-kubelet-client" certificate and key
I0725 19:33:35.222717 698181 kubeadm.go:310] [certs] Generating "front-proxy-ca" certificate and key
I0725 19:33:35.533480 698181 kubeadm.go:310] [certs] Generating "front-proxy-client" certificate and key
I0725 19:33:35.855215 698181 kubeadm.go:310] [certs] Generating "etcd/ca" certificate and key
I0725 19:33:36.276151 698181 kubeadm.go:310] [certs] Generating "etcd/server" certificate and key
I0725 19:33:36.276426 698181 kubeadm.go:310] [certs] etcd/server serving cert is signed for DNS names [embed-certs-240166 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
I0725 19:33:33.236475 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:35.236629 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:36.711778 698181 kubeadm.go:310] [certs] Generating "etcd/peer" certificate and key
I0725 19:33:36.712146 698181 kubeadm.go:310] [certs] etcd/peer serving cert is signed for DNS names [embed-certs-240166 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
I0725 19:33:38.412683 698181 kubeadm.go:310] [certs] Generating "etcd/healthcheck-client" certificate and key
I0725 19:33:38.598281 698181 kubeadm.go:310] [certs] Generating "apiserver-etcd-client" certificate and key
I0725 19:33:39.122498 698181 kubeadm.go:310] [certs] Generating "sa" key and public key
I0725 19:33:39.122857 698181 kubeadm.go:310] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0725 19:33:39.649046 698181 kubeadm.go:310] [kubeconfig] Writing "admin.conf" kubeconfig file
I0725 19:33:39.974265 698181 kubeadm.go:310] [kubeconfig] Writing "super-admin.conf" kubeconfig file
I0725 19:33:40.786699 698181 kubeadm.go:310] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0725 19:33:37.237325 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:39.239015 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:41.737833 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:41.440370 698181 kubeadm.go:310] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0725 19:33:42.261264 698181 kubeadm.go:310] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0725 19:33:42.262035 698181 kubeadm.go:310] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0725 19:33:42.265797 698181 kubeadm.go:310] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0725 19:33:42.268366 698181 out.go:204] - Booting up control plane ...
I0725 19:33:42.268477 698181 kubeadm.go:310] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0725 19:33:42.268561 698181 kubeadm.go:310] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0725 19:33:42.270556 698181 kubeadm.go:310] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0725 19:33:42.286179 698181 kubeadm.go:310] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0725 19:33:42.287459 698181 kubeadm.go:310] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0725 19:33:42.287548 698181 kubeadm.go:310] [kubelet-start] Starting the kubelet
I0725 19:33:42.396216 698181 kubeadm.go:310] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests"
I0725 19:33:42.396307 698181 kubeadm.go:310] [kubelet-check] Waiting for a healthy kubelet. This can take up to 4m0s
I0725 19:33:44.396440 698181 kubeadm.go:310] [kubelet-check] The kubelet is healthy after 2.001397654s
I0725 19:33:44.396528 698181 kubeadm.go:310] [api-check] Waiting for a healthy API server. This can take up to 4m0s
I0725 19:33:44.238271 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:46.737113 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:51.397991 698181 kubeadm.go:310] [api-check] The API server is healthy after 7.001867326s
I0725 19:33:51.421760 698181 kubeadm.go:310] [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
I0725 19:33:51.440506 698181 kubeadm.go:310] [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
I0725 19:33:51.471059 698181 kubeadm.go:310] [upload-certs] Skipping phase. Please see --upload-certs
I0725 19:33:51.471259 698181 kubeadm.go:310] [mark-control-plane] Marking the node embed-certs-240166 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
I0725 19:33:51.485311 698181 kubeadm.go:310] [bootstrap-token] Using token: rhjkyi.ypg72ptr5gntjngs
I0725 19:33:49.236415 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:51.736605 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:51.488104 698181 out.go:204] - Configuring RBAC rules ...
I0725 19:33:51.488236 698181 kubeadm.go:310] [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
I0725 19:33:51.493394 698181 kubeadm.go:310] [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
I0725 19:33:51.502061 698181 kubeadm.go:310] [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
I0725 19:33:51.506835 698181 kubeadm.go:310] [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
I0725 19:33:51.511184 698181 kubeadm.go:310] [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
I0725 19:33:51.517454 698181 kubeadm.go:310] [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
I0725 19:33:51.805736 698181 kubeadm.go:310] [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
I0725 19:33:52.237957 698181 kubeadm.go:310] [addons] Applied essential addon: CoreDNS
I0725 19:33:52.805032 698181 kubeadm.go:310] [addons] Applied essential addon: kube-proxy
I0725 19:33:52.806244 698181 kubeadm.go:310]
I0725 19:33:52.806318 698181 kubeadm.go:310] Your Kubernetes control-plane has initialized successfully!
I0725 19:33:52.806325 698181 kubeadm.go:310]
I0725 19:33:52.806399 698181 kubeadm.go:310] To start using your cluster, you need to run the following as a regular user:
I0725 19:33:52.806403 698181 kubeadm.go:310]
I0725 19:33:52.806428 698181 kubeadm.go:310] mkdir -p $HOME/.kube
I0725 19:33:52.806493 698181 kubeadm.go:310] sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
I0725 19:33:52.806545 698181 kubeadm.go:310] sudo chown $(id -u):$(id -g) $HOME/.kube/config
I0725 19:33:52.806552 698181 kubeadm.go:310]
I0725 19:33:52.806613 698181 kubeadm.go:310] Alternatively, if you are the root user, you can run:
I0725 19:33:52.806618 698181 kubeadm.go:310]
I0725 19:33:52.806664 698181 kubeadm.go:310] export KUBECONFIG=/etc/kubernetes/admin.conf
I0725 19:33:52.806669 698181 kubeadm.go:310]
I0725 19:33:52.806719 698181 kubeadm.go:310] You should now deploy a pod network to the cluster.
I0725 19:33:52.806790 698181 kubeadm.go:310] Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
I0725 19:33:52.806856 698181 kubeadm.go:310] https://kubernetes.io/docs/concepts/cluster-administration/addons/
I0725 19:33:52.806860 698181 kubeadm.go:310]
I0725 19:33:52.806974 698181 kubeadm.go:310] You can now join any number of control-plane nodes by copying certificate authorities
I0725 19:33:52.807050 698181 kubeadm.go:310] and service account keys on each node and then running the following as root:
I0725 19:33:52.807055 698181 kubeadm.go:310]
I0725 19:33:52.807135 698181 kubeadm.go:310] kubeadm join control-plane.minikube.internal:8443 --token rhjkyi.ypg72ptr5gntjngs \
I0725 19:33:52.807234 698181 kubeadm.go:310] --discovery-token-ca-cert-hash sha256:1d9d423c7b2bb24c3d3f38c79b211d8531f40310f79a51ed9602d13a57b81b8c \
I0725 19:33:52.807254 698181 kubeadm.go:310] --control-plane
I0725 19:33:52.807259 698181 kubeadm.go:310]
I0725 19:33:52.807340 698181 kubeadm.go:310] Then you can join any number of worker nodes by running the following on each as root:
I0725 19:33:52.807345 698181 kubeadm.go:310]
I0725 19:33:52.807423 698181 kubeadm.go:310] kubeadm join control-plane.minikube.internal:8443 --token rhjkyi.ypg72ptr5gntjngs \
I0725 19:33:52.807522 698181 kubeadm.go:310] --discovery-token-ca-cert-hash sha256:1d9d423c7b2bb24c3d3f38c79b211d8531f40310f79a51ed9602d13a57b81b8c
I0725 19:33:52.811047 698181 kubeadm.go:310] [WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1065-aws\n", err: exit status 1
I0725 19:33:52.811183 698181 kubeadm.go:310] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0725 19:33:52.811204 698181 cni.go:84] Creating CNI manager for ""
I0725 19:33:52.811221 698181 cni.go:143] "docker" driver + "containerd" runtime found, recommending kindnet
I0725 19:33:52.814176 698181 out.go:177] * Configuring CNI (Container Networking Interface) ...
I0725 19:33:52.816754 698181 ssh_runner.go:195] Run: stat /opt/cni/bin/portmap
I0725 19:33:52.821663 698181 cni.go:182] applying CNI manifest using /var/lib/minikube/binaries/v1.30.3/kubectl ...
I0725 19:33:52.821686 698181 ssh_runner.go:362] scp memory --> /var/tmp/minikube/cni.yaml (2438 bytes)
I0725 19:33:52.841213 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl apply --kubeconfig=/var/lib/minikube/kubeconfig -f /var/tmp/minikube/cni.yaml
I0725 19:33:53.116641 698181 ssh_runner.go:195] Run: /bin/bash -c "cat /proc/$(pgrep kube-apiserver)/oom_adj"
I0725 19:33:53.116796 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl create clusterrolebinding minikube-rbac --clusterrole=cluster-admin --serviceaccount=kube-system:default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:33:53.116912 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl --kubeconfig=/var/lib/minikube/kubeconfig label --overwrite nodes embed-certs-240166 minikube.k8s.io/updated_at=2024_07_25T19_33_53_0700 minikube.k8s.io/version=v1.33.1 minikube.k8s.io/commit=6f15797740a09e0fb947959f5fd09f2e323bde5a minikube.k8s.io/name=embed-certs-240166 minikube.k8s.io/primary=true
I0725 19:33:53.323222 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:33:53.323233 698181 ops.go:34] apiserver oom_adj: -16
I0725 19:33:53.824048 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:33:54.323325 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:33:54.823565 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:33:55.323386 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:33:55.824196 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:33:54.236461 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:56.735944 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:33:56.323578 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:33:56.823306 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:33:57.323389 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:33:57.823867 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:33:58.323477 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:33:58.824357 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:33:59.324142 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:33:59.823401 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:34:00.325200 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:34:00.824053 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:33:59.235650 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:01.736227 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:01.324347 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:34:01.824176 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:34:02.324171 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:34:02.824048 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:34:03.324129 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:34:03.824184 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:34:04.323594 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:34:04.824151 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:34:05.324176 698181 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.30.3/kubectl get sa default --kubeconfig=/var/lib/minikube/kubeconfig
I0725 19:34:05.434803 698181 kubeadm.go:1113] duration metric: took 12.318066159s to wait for elevateKubeSystemPrivileges
I0725 19:34:05.434839 698181 kubeadm.go:394] duration metric: took 31.606218155s to StartCluster
I0725 19:34:05.434857 698181 settings.go:142] acquiring lock: {Name:mk69edff96840eebb76289a50cf78daf601fe5de Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0725 19:34:05.434923 698181 settings.go:150] Updating kubeconfig: /home/jenkins/minikube-integration/19326-431487/kubeconfig
I0725 19:34:05.436318 698181 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/19326-431487/kubeconfig: {Name:mk3cdfe1101bbbc0f7441d92cff5cd6b29ee3404 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0725 19:34:05.436564 698181 start.go:235] Will wait 6m0s for node &{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.30.3 ContainerRuntime:containerd ControlPlane:true Worker:true}
I0725 19:34:05.436723 698181 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.30.3/kubectl --kubeconfig=/var/lib/minikube/kubeconfig -n kube-system get configmap coredns -o yaml"
I0725 19:34:05.436984 698181 config.go:182] Loaded profile config "embed-certs-240166": Driver=docker, ContainerRuntime=containerd, KubernetesVersion=v1.30.3
I0725 19:34:05.437028 698181 addons.go:507] enable addons start: toEnable=map[ambassador:false auto-pause:false cloud-spanner:false csi-hostpath-driver:false dashboard:false default-storageclass:true efk:false freshpod:false gcp-auth:false gvisor:false headlamp:false helm-tiller:false inaccel:false ingress:false ingress-dns:false inspektor-gadget:false istio:false istio-provisioner:false kong:false kubeflow:false kubevirt:false logviewer:false metallb:false metrics-server:false nvidia-device-plugin:false nvidia-driver-installer:false nvidia-gpu-device-plugin:false olm:false pod-security-policy:false portainer:false registry:false registry-aliases:false registry-creds:false storage-provisioner:true storage-provisioner-gluster:false storage-provisioner-rancher:false volcano:false volumesnapshots:false yakd:false]
I0725 19:34:05.437094 698181 addons.go:69] Setting storage-provisioner=true in profile "embed-certs-240166"
I0725 19:34:05.437116 698181 addons.go:234] Setting addon storage-provisioner=true in "embed-certs-240166"
I0725 19:34:05.437139 698181 host.go:66] Checking if "embed-certs-240166" exists ...
I0725 19:34:05.437827 698181 cli_runner.go:164] Run: docker container inspect embed-certs-240166 --format={{.State.Status}}
I0725 19:34:05.437896 698181 addons.go:69] Setting default-storageclass=true in profile "embed-certs-240166"
I0725 19:34:05.437930 698181 addons_storage_classes.go:33] enableOrDisableStorageClasses default-storageclass=true on "embed-certs-240166"
I0725 19:34:05.438154 698181 cli_runner.go:164] Run: docker container inspect embed-certs-240166 --format={{.State.Status}}
I0725 19:34:05.440393 698181 out.go:177] * Verifying Kubernetes components...
I0725 19:34:05.443237 698181 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0725 19:34:05.472212 698181 addons.go:234] Setting addon default-storageclass=true in "embed-certs-240166"
I0725 19:34:05.472255 698181 host.go:66] Checking if "embed-certs-240166" exists ...
I0725 19:34:05.472673 698181 cli_runner.go:164] Run: docker container inspect embed-certs-240166 --format={{.State.Status}}
I0725 19:34:05.481196 698181 out.go:177] - Using image gcr.io/k8s-minikube/storage-provisioner:v5
I0725 19:34:05.483953 698181 addons.go:431] installing /etc/kubernetes/addons/storage-provisioner.yaml
I0725 19:34:05.483972 698181 ssh_runner.go:362] scp memory --> /etc/kubernetes/addons/storage-provisioner.yaml (2676 bytes)
I0725 19:34:05.484040 698181 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-240166
I0725 19:34:05.516136 698181 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33506 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/embed-certs-240166/id_rsa Username:docker}
I0725 19:34:05.518612 698181 addons.go:431] installing /etc/kubernetes/addons/storageclass.yaml
I0725 19:34:05.518634 698181 ssh_runner.go:362] scp storageclass/storageclass.yaml --> /etc/kubernetes/addons/storageclass.yaml (271 bytes)
I0725 19:34:05.518705 698181 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" embed-certs-240166
I0725 19:34:05.540433 698181 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33506 SSHKeyPath:/home/jenkins/minikube-integration/19326-431487/.minikube/machines/embed-certs-240166/id_rsa Username:docker}
I0725 19:34:05.908291 698181 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.30.3/kubectl apply -f /etc/kubernetes/addons/storageclass.yaml
I0725 19:34:05.931484 698181 ssh_runner.go:195] Run: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.30.3/kubectl apply -f /etc/kubernetes/addons/storage-provisioner.yaml
I0725 19:34:05.947466 698181 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.30.3/kubectl --kubeconfig=/var/lib/minikube/kubeconfig -n kube-system get configmap coredns -o yaml | sed -e '/^ forward . \/etc\/resolv.conf.*/i \ hosts {\n 192.168.76.1 host.minikube.internal\n fallthrough\n }' -e '/^ errors *$/i \ log' | sudo /var/lib/minikube/binaries/v1.30.3/kubectl --kubeconfig=/var/lib/minikube/kubeconfig replace -f -"
I0725 19:34:05.947588 698181 ssh_runner.go:195] Run: sudo systemctl start kubelet
I0725 19:34:03.736798 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:06.235979 688921 pod_ready.go:102] pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:06.737497 688921 pod_ready.go:81] duration metric: took 4m0.008287476s for pod "metrics-server-9975d5f86-8g9gp" in "kube-system" namespace to be "Ready" ...
E0725 19:34:06.737526 688921 pod_ready.go:66] WaitExtra: waitPodCondition: context deadline exceeded
I0725 19:34:06.737535 688921 pod_ready.go:38] duration metric: took 5m24.79684369s for extra waiting for all system-critical and pods with labels [k8s-app=kube-dns component=etcd component=kube-apiserver component=kube-controller-manager k8s-app=kube-proxy component=kube-scheduler] to be "Ready" ...
I0725 19:34:06.737548 688921 api_server.go:52] waiting for apiserver process to appear ...
I0725 19:34:06.737576 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0725 19:34:06.737650 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0725 19:34:06.956609 698181 ssh_runner.go:235] Completed: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.30.3/kubectl apply -f /etc/kubernetes/addons/storage-provisioner.yaml: (1.025090111s)
I0725 19:34:06.956817 698181 ssh_runner.go:235] Completed: sudo systemctl start kubelet: (1.00921085s)
I0725 19:34:06.957045 698181 ssh_runner.go:235] Completed: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.30.3/kubectl --kubeconfig=/var/lib/minikube/kubeconfig -n kube-system get configmap coredns -o yaml | sed -e '/^ forward . \/etc\/resolv.conf.*/i \ hosts {\n 192.168.76.1 host.minikube.internal\n fallthrough\n }' -e '/^ errors *$/i \ log' | sudo /var/lib/minikube/binaries/v1.30.3/kubectl --kubeconfig=/var/lib/minikube/kubeconfig replace -f -": (1.009550961s)
I0725 19:34:06.957063 698181 start.go:971] {"host.minikube.internal": 192.168.76.1} host record injected into CoreDNS's ConfigMap
I0725 19:34:06.960334 698181 out.go:177] * Enabled addons: default-storageclass, storage-provisioner
I0725 19:34:06.960601 698181 node_ready.go:35] waiting up to 6m0s for node "embed-certs-240166" to be "Ready" ...
I0725 19:34:06.963201 698181 addons.go:510] duration metric: took 1.526164291s for enable addons: enabled=[default-storageclass storage-provisioner]
I0725 19:34:06.983393 698181 node_ready.go:49] node "embed-certs-240166" has status "Ready":"True"
I0725 19:34:06.983419 698181 node_ready.go:38] duration metric: took 22.764089ms for node "embed-certs-240166" to be "Ready" ...
I0725 19:34:06.983430 698181 pod_ready.go:35] extra waiting up to 6m0s for all system-critical pods including labels [k8s-app=kube-dns component=etcd component=kube-apiserver component=kube-controller-manager k8s-app=kube-proxy component=kube-scheduler] to be "Ready" ...
I0725 19:34:07.004487 698181 pod_ready.go:78] waiting up to 6m0s for pod "coredns-7db6d8ff4d-2dw2z" in "kube-system" namespace to be "Ready" ...
I0725 19:34:07.469766 698181 kapi.go:214] "coredns" deployment in "kube-system" namespace and "embed-certs-240166" context rescaled to 1 replicas
I0725 19:34:09.012241 698181 pod_ready.go:102] pod "coredns-7db6d8ff4d-2dw2z" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:06.807692 688921 cri.go:89] found id: "8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7"
I0725 19:34:06.807723 688921 cri.go:89] found id: "a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8"
I0725 19:34:06.807729 688921 cri.go:89] found id: ""
I0725 19:34:06.807736 688921 logs.go:276] 2 containers: [8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7 a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8]
I0725 19:34:06.807794 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:06.818733 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:06.822775 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0725 19:34:06.822852 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0725 19:34:06.888510 688921 cri.go:89] found id: "1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967"
I0725 19:34:06.888542 688921 cri.go:89] found id: "bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6"
I0725 19:34:06.888547 688921 cri.go:89] found id: ""
I0725 19:34:06.888557 688921 logs.go:276] 2 containers: [1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967 bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6]
I0725 19:34:06.888612 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:06.892832 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:06.897974 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0725 19:34:06.898094 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0725 19:34:06.959812 688921 cri.go:89] found id: "aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05"
I0725 19:34:06.959838 688921 cri.go:89] found id: "6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8"
I0725 19:34:06.959844 688921 cri.go:89] found id: ""
I0725 19:34:06.959851 688921 logs.go:276] 2 containers: [aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05 6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8]
I0725 19:34:06.959956 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:06.964200 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:06.968411 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0725 19:34:06.968494 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0725 19:34:07.033397 688921 cri.go:89] found id: "fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1"
I0725 19:34:07.033436 688921 cri.go:89] found id: "2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34"
I0725 19:34:07.033442 688921 cri.go:89] found id: ""
I0725 19:34:07.033450 688921 logs.go:276] 2 containers: [fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1 2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34]
I0725 19:34:07.033516 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.043063 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.048413 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0725 19:34:07.048505 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0725 19:34:07.111832 688921 cri.go:89] found id: "09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98"
I0725 19:34:07.111912 688921 cri.go:89] found id: "5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f"
I0725 19:34:07.111931 688921 cri.go:89] found id: ""
I0725 19:34:07.111952 688921 logs.go:276] 2 containers: [09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98 5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f]
I0725 19:34:07.112033 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.117008 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.121337 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0725 19:34:07.121457 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0725 19:34:07.187319 688921 cri.go:89] found id: "83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5"
I0725 19:34:07.187384 688921 cri.go:89] found id: "092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb"
I0725 19:34:07.187401 688921 cri.go:89] found id: ""
I0725 19:34:07.187423 688921 logs.go:276] 2 containers: [83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5 092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb]
I0725 19:34:07.187507 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.192267 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.196794 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kindnet Namespaces:[]}
I0725 19:34:07.196914 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kindnet
I0725 19:34:07.271398 688921 cri.go:89] found id: "ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01"
I0725 19:34:07.271471 688921 cri.go:89] found id: "687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8"
I0725 19:34:07.271489 688921 cri.go:89] found id: ""
I0725 19:34:07.271511 688921 logs.go:276] 2 containers: [ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01 687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8]
I0725 19:34:07.271595 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.275933 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.281359 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0725 19:34:07.281487 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0725 19:34:07.377349 688921 cri.go:89] found id: "1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e"
I0725 19:34:07.377413 688921 cri.go:89] found id: "6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985"
I0725 19:34:07.377432 688921 cri.go:89] found id: ""
I0725 19:34:07.377454 688921 logs.go:276] 2 containers: [1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e 6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985]
I0725 19:34:07.377542 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.383264 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.388959 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0725 19:34:07.389087 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0725 19:34:07.461197 688921 cri.go:89] found id: "22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc"
I0725 19:34:07.461216 688921 cri.go:89] found id: ""
I0725 19:34:07.461223 688921 logs.go:276] 1 containers: [22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc]
I0725 19:34:07.461279 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:07.465404 688921 logs.go:123] Gathering logs for storage-provisioner [1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e] ...
I0725 19:34:07.465428 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e"
I0725 19:34:07.547164 688921 logs.go:123] Gathering logs for container status ...
I0725 19:34:07.547233 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0725 19:34:07.624426 688921 logs.go:123] Gathering logs for kubelet ...
I0725 19:34:07.624507 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0725 19:34:07.703403 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:42 old-k8s-version-262689 kubelet[657]: E0725 19:28:42.036852 657 reflector.go:138] object-"kube-system"/"kindnet-token-vxqld": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kindnet-token-vxqld" is forbidden: User "system:node:old-k8s-version-262689" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-262689' and this object
W0725 19:34:07.703640 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:42 old-k8s-version-262689 kubelet[657]: E0725 19:28:42.037214 657 reflector.go:138] object-"kube-system"/"kube-proxy-token-2gxrg": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kube-proxy-token-2gxrg" is forbidden: User "system:node:old-k8s-version-262689" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-262689' and this object
W0725 19:34:07.707435 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:45 old-k8s-version-262689 kubelet[657]: E0725 19:28:45.149554 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:07.707634 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:46 old-k8s-version-262689 kubelet[657]: E0725 19:28:46.003537 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.710803 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:57 old-k8s-version-262689 kubelet[657]: E0725 19:28:57.257602 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:07.713278 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:09 old-k8s-version-262689 kubelet[657]: E0725 19:29:09.369957 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.713674 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:10 old-k8s-version-262689 kubelet[657]: E0725 19:29:10.383955 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.713885 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:11 old-k8s-version-262689 kubelet[657]: E0725 19:29:11.215410 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.714611 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:16 old-k8s-version-262689 kubelet[657]: E0725 19:29:16.271158 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.715106 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:17 old-k8s-version-262689 kubelet[657]: E0725 19:29:17.406387 657 pod_workers.go:191] Error syncing pod 192a4c32-53cd-4ce7-ba80-a523469b645d ("storage-provisioner_kube-system(192a4c32-53cd-4ce7-ba80-a523469b645d)"), skipping: failed to "StartContainer" for "storage-provisioner" with CrashLoopBackOff: "back-off 10s restarting failed container=storage-provisioner pod=storage-provisioner_kube-system(192a4c32-53cd-4ce7-ba80-a523469b645d)"
W0725 19:34:07.718311 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:26 old-k8s-version-262689 kubelet[657]: E0725 19:29:26.224127 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:07.719087 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:29 old-k8s-version-262689 kubelet[657]: E0725 19:29:29.440381 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.719629 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:36 old-k8s-version-262689 kubelet[657]: E0725 19:29:36.271666 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.719827 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:37 old-k8s-version-262689 kubelet[657]: E0725 19:29:37.215365 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.720195 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:51 old-k8s-version-262689 kubelet[657]: E0725 19:29:51.215710 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.720693 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:51 old-k8s-version-262689 kubelet[657]: E0725 19:29:51.524343 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.721057 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:56 old-k8s-version-262689 kubelet[657]: E0725 19:29:56.271047 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.721244 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:04 old-k8s-version-262689 kubelet[657]: E0725 19:30:04.215943 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.721572 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:11 old-k8s-version-262689 kubelet[657]: E0725 19:30:11.215709 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.724232 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:19 old-k8s-version-262689 kubelet[657]: E0725 19:30:19.231937 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:07.724588 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:25 old-k8s-version-262689 kubelet[657]: E0725 19:30:25.214643 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.724789 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:32 old-k8s-version-262689 kubelet[657]: E0725 19:30:32.216336 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.725458 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:40 old-k8s-version-262689 kubelet[657]: E0725 19:30:40.656202 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.725659 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:46 old-k8s-version-262689 kubelet[657]: E0725 19:30:46.215221 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.726057 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:46 old-k8s-version-262689 kubelet[657]: E0725 19:30:46.271828 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.726400 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:01 old-k8s-version-262689 kubelet[657]: E0725 19:31:01.215458 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.726633 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:01 old-k8s-version-262689 kubelet[657]: E0725 19:31:01.215611 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.726841 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:13 old-k8s-version-262689 kubelet[657]: E0725 19:31:13.215172 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.727236 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:14 old-k8s-version-262689 kubelet[657]: E0725 19:31:14.214851 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.727622 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:26 old-k8s-version-262689 kubelet[657]: E0725 19:31:26.215358 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.727848 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:27 old-k8s-version-262689 kubelet[657]: E0725 19:31:27.215041 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.728224 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:37 old-k8s-version-262689 kubelet[657]: E0725 19:31:37.215248 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.731097 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:40 old-k8s-version-262689 kubelet[657]: E0725 19:31:40.223791 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:07.731472 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:52 old-k8s-version-262689 kubelet[657]: E0725 19:31:52.217037 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.731662 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:54 old-k8s-version-262689 kubelet[657]: E0725 19:31:54.225403 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.731924 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:05 old-k8s-version-262689 kubelet[657]: E0725 19:32:05.215333 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.732602 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:07 old-k8s-version-262689 kubelet[657]: E0725 19:32:07.888105 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.732942 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:16 old-k8s-version-262689 kubelet[657]: E0725 19:32:16.271463 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.733127 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:19 old-k8s-version-262689 kubelet[657]: E0725 19:32:19.215018 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.733595 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:28 old-k8s-version-262689 kubelet[657]: E0725 19:32:28.214662 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.733787 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:31 old-k8s-version-262689 kubelet[657]: E0725 19:32:31.215144 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.734174 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:41 old-k8s-version-262689 kubelet[657]: E0725 19:32:41.214882 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.734408 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:46 old-k8s-version-262689 kubelet[657]: E0725 19:32:46.215198 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.734844 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:52 old-k8s-version-262689 kubelet[657]: E0725 19:32:52.215208 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.735085 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:59 old-k8s-version-262689 kubelet[657]: E0725 19:32:59.215146 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.735513 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:06 old-k8s-version-262689 kubelet[657]: E0725 19:33:06.214773 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.735733 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:12 old-k8s-version-262689 kubelet[657]: E0725 19:33:12.219911 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.736070 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:17 old-k8s-version-262689 kubelet[657]: E0725 19:33:17.215219 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.736255 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:25 old-k8s-version-262689 kubelet[657]: E0725 19:33:25.215021 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.736581 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:32 old-k8s-version-262689 kubelet[657]: E0725 19:33:32.216211 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.736764 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:38 old-k8s-version-262689 kubelet[657]: E0725 19:33:38.215054 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.737090 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:45 old-k8s-version-262689 kubelet[657]: E0725 19:33:45.216141 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.737377 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:50 old-k8s-version-262689 kubelet[657]: E0725 19:33:50.215161 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:07.737718 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:58 old-k8s-version-262689 kubelet[657]: E0725 19:33:58.214660 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:07.737917 688921 logs.go:138] Found kubelet problem: Jul 25 19:34:03 old-k8s-version-262689 kubelet[657]: E0725 19:34:03.215098 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
I0725 19:34:07.737927 688921 logs.go:123] Gathering logs for etcd [1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967] ...
I0725 19:34:07.737943 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967"
I0725 19:34:07.822664 688921 logs.go:123] Gathering logs for kube-scheduler [fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1] ...
I0725 19:34:07.822740 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1"
I0725 19:34:07.879095 688921 logs.go:123] Gathering logs for kindnet [ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01] ...
I0725 19:34:07.879168 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01"
I0725 19:34:07.955648 688921 logs.go:123] Gathering logs for kindnet [687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8] ...
I0725 19:34:07.955730 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8"
I0725 19:34:08.037629 688921 logs.go:123] Gathering logs for kube-apiserver [8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7] ...
I0725 19:34:08.037704 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7"
I0725 19:34:08.114974 688921 logs.go:123] Gathering logs for kube-scheduler [2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34] ...
I0725 19:34:08.115040 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34"
I0725 19:34:08.198695 688921 logs.go:123] Gathering logs for kube-controller-manager [83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5] ...
I0725 19:34:08.198774 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5"
I0725 19:34:08.332924 688921 logs.go:123] Gathering logs for kube-controller-manager [092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb] ...
I0725 19:34:08.333002 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb"
I0725 19:34:08.461781 688921 logs.go:123] Gathering logs for kube-proxy [5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f] ...
I0725 19:34:08.461879 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f"
I0725 19:34:08.540031 688921 logs.go:123] Gathering logs for storage-provisioner [6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985] ...
I0725 19:34:08.540060 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985"
I0725 19:34:08.583359 688921 logs.go:123] Gathering logs for containerd ...
I0725 19:34:08.583437 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0725 19:34:08.659982 688921 logs.go:123] Gathering logs for dmesg ...
I0725 19:34:08.660061 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0725 19:34:08.680183 688921 logs.go:123] Gathering logs for kube-apiserver [a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8] ...
I0725 19:34:08.680216 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8"
I0725 19:34:08.758790 688921 logs.go:123] Gathering logs for coredns [6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8] ...
I0725 19:34:08.758823 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8"
I0725 19:34:08.810366 688921 logs.go:123] Gathering logs for kube-proxy [09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98] ...
I0725 19:34:08.810401 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98"
I0725 19:34:08.855821 688921 logs.go:123] Gathering logs for describe nodes ...
I0725 19:34:08.855850 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.20.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
I0725 19:34:09.049486 688921 logs.go:123] Gathering logs for etcd [bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6] ...
I0725 19:34:09.049516 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6"
I0725 19:34:09.101862 688921 logs.go:123] Gathering logs for coredns [aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05] ...
I0725 19:34:09.101895 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05"
I0725 19:34:09.150649 688921 logs.go:123] Gathering logs for kubernetes-dashboard [22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc] ...
I0725 19:34:09.150683 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc"
I0725 19:34:09.195050 688921 out.go:304] Setting ErrFile to fd 2...
I0725 19:34:09.195080 688921 out.go:338] TERM=,COLORTERM=, which probably does not support color
W0725 19:34:09.195179 688921 out.go:239] X Problems detected in kubelet:
W0725 19:34:09.195327 688921 out.go:239] Jul 25 19:33:38 old-k8s-version-262689 kubelet[657]: E0725 19:33:38.215054 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:09.195345 688921 out.go:239] Jul 25 19:33:45 old-k8s-version-262689 kubelet[657]: E0725 19:33:45.216141 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:09.195424 688921 out.go:239] Jul 25 19:33:50 old-k8s-version-262689 kubelet[657]: E0725 19:33:50.215161 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:09.195440 688921 out.go:239] Jul 25 19:33:58 old-k8s-version-262689 kubelet[657]: E0725 19:33:58.214660 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:09.195465 688921 out.go:239] Jul 25 19:34:03 old-k8s-version-262689 kubelet[657]: E0725 19:34:03.215098 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
I0725 19:34:09.195473 688921 out.go:304] Setting ErrFile to fd 2...
I0725 19:34:09.195484 688921 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0725 19:34:11.511518 698181 pod_ready.go:102] pod "coredns-7db6d8ff4d-2dw2z" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:13.511969 698181 pod_ready.go:102] pod "coredns-7db6d8ff4d-2dw2z" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:16.013214 698181 pod_ready.go:102] pod "coredns-7db6d8ff4d-2dw2z" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:18.014038 698181 pod_ready.go:102] pod "coredns-7db6d8ff4d-2dw2z" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:20.512482 698181 pod_ready.go:102] pod "coredns-7db6d8ff4d-2dw2z" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:19.196744 688921 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0725 19:34:19.209410 688921 api_server.go:72] duration metric: took 5m55.042327442s to wait for apiserver process to appear ...
I0725 19:34:19.209435 688921 api_server.go:88] waiting for apiserver healthz status ...
I0725 19:34:19.209472 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0725 19:34:19.209531 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0725 19:34:19.247922 688921 cri.go:89] found id: "8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7"
I0725 19:34:19.247942 688921 cri.go:89] found id: "a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8"
I0725 19:34:19.247947 688921 cri.go:89] found id: ""
I0725 19:34:19.247954 688921 logs.go:276] 2 containers: [8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7 a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8]
I0725 19:34:19.248012 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.252109 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.255778 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0725 19:34:19.255850 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0725 19:34:19.295831 688921 cri.go:89] found id: "1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967"
I0725 19:34:19.295853 688921 cri.go:89] found id: "bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6"
I0725 19:34:19.295859 688921 cri.go:89] found id: ""
I0725 19:34:19.295866 688921 logs.go:276] 2 containers: [1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967 bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6]
I0725 19:34:19.295924 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.300077 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.303811 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0725 19:34:19.303883 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0725 19:34:19.349146 688921 cri.go:89] found id: "aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05"
I0725 19:34:19.349170 688921 cri.go:89] found id: "6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8"
I0725 19:34:19.349176 688921 cri.go:89] found id: ""
I0725 19:34:19.349183 688921 logs.go:276] 2 containers: [aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05 6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8]
I0725 19:34:19.349247 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.353162 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.356919 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0725 19:34:19.357009 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0725 19:34:19.396994 688921 cri.go:89] found id: "fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1"
I0725 19:34:19.397014 688921 cri.go:89] found id: "2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34"
I0725 19:34:19.397018 688921 cri.go:89] found id: ""
I0725 19:34:19.397025 688921 logs.go:276] 2 containers: [fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1 2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34]
I0725 19:34:19.397084 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.401204 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.405026 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0725 19:34:19.405158 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0725 19:34:19.446347 688921 cri.go:89] found id: "09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98"
I0725 19:34:19.446370 688921 cri.go:89] found id: "5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f"
I0725 19:34:19.446375 688921 cri.go:89] found id: ""
I0725 19:34:19.446383 688921 logs.go:276] 2 containers: [09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98 5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f]
I0725 19:34:19.446443 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.450086 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.453624 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0725 19:34:19.453731 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0725 19:34:19.492956 688921 cri.go:89] found id: "83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5"
I0725 19:34:19.492981 688921 cri.go:89] found id: "092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb"
I0725 19:34:19.492985 688921 cri.go:89] found id: ""
I0725 19:34:19.492993 688921 logs.go:276] 2 containers: [83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5 092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb]
I0725 19:34:19.493051 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.497081 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.500953 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kindnet Namespaces:[]}
I0725 19:34:19.501052 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kindnet
I0725 19:34:19.546766 688921 cri.go:89] found id: "ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01"
I0725 19:34:19.546791 688921 cri.go:89] found id: "687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8"
I0725 19:34:19.546797 688921 cri.go:89] found id: ""
I0725 19:34:19.546804 688921 logs.go:276] 2 containers: [ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01 687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8]
I0725 19:34:19.546860 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.550701 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.554230 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0725 19:34:19.554305 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0725 19:34:19.594222 688921 cri.go:89] found id: "1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e"
I0725 19:34:19.594256 688921 cri.go:89] found id: "6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985"
I0725 19:34:19.594261 688921 cri.go:89] found id: ""
I0725 19:34:19.594269 688921 logs.go:276] 2 containers: [1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e 6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985]
I0725 19:34:19.594337 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.598412 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.602080 688921 cri.go:54] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0725 19:34:19.602161 688921 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0725 19:34:19.643928 688921 cri.go:89] found id: "22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc"
I0725 19:34:19.643955 688921 cri.go:89] found id: ""
I0725 19:34:19.643963 688921 logs.go:276] 1 containers: [22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc]
I0725 19:34:19.644027 688921 ssh_runner.go:195] Run: which crictl
I0725 19:34:19.647751 688921 logs.go:123] Gathering logs for container status ...
I0725 19:34:19.647778 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0725 19:34:19.690394 688921 logs.go:123] Gathering logs for etcd [1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967] ...
I0725 19:34:19.690510 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967"
I0725 19:34:19.732117 688921 logs.go:123] Gathering logs for kube-proxy [09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98] ...
I0725 19:34:19.732146 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98"
I0725 19:34:19.780472 688921 logs.go:123] Gathering logs for kube-controller-manager [83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5] ...
I0725 19:34:19.780499 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5"
I0725 19:34:19.851807 688921 logs.go:123] Gathering logs for kindnet [687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8] ...
I0725 19:34:19.851890 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8"
I0725 19:34:19.909403 688921 logs.go:123] Gathering logs for storage-provisioner [1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e] ...
I0725 19:34:19.909441 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e"
I0725 19:34:19.947085 688921 logs.go:123] Gathering logs for kubernetes-dashboard [22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc] ...
I0725 19:34:19.947112 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc"
I0725 19:34:19.989416 688921 logs.go:123] Gathering logs for kindnet [ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01] ...
I0725 19:34:19.989441 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01"
I0725 19:34:20.080244 688921 logs.go:123] Gathering logs for containerd ...
I0725 19:34:20.080284 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0725 19:34:20.148363 688921 logs.go:123] Gathering logs for kubelet ...
I0725 19:34:20.148399 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0725 19:34:20.209971 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:42 old-k8s-version-262689 kubelet[657]: E0725 19:28:42.036852 657 reflector.go:138] object-"kube-system"/"kindnet-token-vxqld": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kindnet-token-vxqld" is forbidden: User "system:node:old-k8s-version-262689" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-262689' and this object
W0725 19:34:20.210213 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:42 old-k8s-version-262689 kubelet[657]: E0725 19:28:42.037214 657 reflector.go:138] object-"kube-system"/"kube-proxy-token-2gxrg": Failed to watch *v1.Secret: failed to list *v1.Secret: secrets "kube-proxy-token-2gxrg" is forbidden: User "system:node:old-k8s-version-262689" cannot list resource "secrets" in API group "" in the namespace "kube-system": no relationship found between node 'old-k8s-version-262689' and this object
W0725 19:34:20.213793 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:45 old-k8s-version-262689 kubelet[657]: E0725 19:28:45.149554 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:20.215174 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:46 old-k8s-version-262689 kubelet[657]: E0725 19:28:46.003537 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.218024 688921 logs.go:138] Found kubelet problem: Jul 25 19:28:57 old-k8s-version-262689 kubelet[657]: E0725 19:28:57.257602 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:20.220231 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:09 old-k8s-version-262689 kubelet[657]: E0725 19:29:09.369957 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.220930 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:10 old-k8s-version-262689 kubelet[657]: E0725 19:29:10.383955 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.221254 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:11 old-k8s-version-262689 kubelet[657]: E0725 19:29:11.215410 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.222117 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:16 old-k8s-version-262689 kubelet[657]: E0725 19:29:16.271158 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 10s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.222720 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:17 old-k8s-version-262689 kubelet[657]: E0725 19:29:17.406387 657 pod_workers.go:191] Error syncing pod 192a4c32-53cd-4ce7-ba80-a523469b645d ("storage-provisioner_kube-system(192a4c32-53cd-4ce7-ba80-a523469b645d)"), skipping: failed to "StartContainer" for "storage-provisioner" with CrashLoopBackOff: "back-off 10s restarting failed container=storage-provisioner pod=storage-provisioner_kube-system(192a4c32-53cd-4ce7-ba80-a523469b645d)"
W0725 19:34:20.226773 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:26 old-k8s-version-262689 kubelet[657]: E0725 19:29:26.224127 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:20.227465 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:29 old-k8s-version-262689 kubelet[657]: E0725 19:29:29.440381 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.227957 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:36 old-k8s-version-262689 kubelet[657]: E0725 19:29:36.271666 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.228162 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:37 old-k8s-version-262689 kubelet[657]: E0725 19:29:37.215365 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.228549 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:51 old-k8s-version-262689 kubelet[657]: E0725 19:29:51.215710 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.229042 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:51 old-k8s-version-262689 kubelet[657]: E0725 19:29:51.524343 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.229493 688921 logs.go:138] Found kubelet problem: Jul 25 19:29:56 old-k8s-version-262689 kubelet[657]: E0725 19:29:56.271047 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.229718 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:04 old-k8s-version-262689 kubelet[657]: E0725 19:30:04.215943 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.230164 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:11 old-k8s-version-262689 kubelet[657]: E0725 19:30:11.215709 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.232845 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:19 old-k8s-version-262689 kubelet[657]: E0725 19:30:19.231937 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:20.233247 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:25 old-k8s-version-262689 kubelet[657]: E0725 19:30:25.214643 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.233529 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:32 old-k8s-version-262689 kubelet[657]: E0725 19:30:32.216336 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.234139 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:40 old-k8s-version-262689 kubelet[657]: E0725 19:30:40.656202 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.234363 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:46 old-k8s-version-262689 kubelet[657]: E0725 19:30:46.215221 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.234728 688921 logs.go:138] Found kubelet problem: Jul 25 19:30:46 old-k8s-version-262689 kubelet[657]: E0725 19:30:46.271828 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.235078 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:01 old-k8s-version-262689 kubelet[657]: E0725 19:31:01.215458 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.235295 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:01 old-k8s-version-262689 kubelet[657]: E0725 19:31:01.215611 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.235532 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:13 old-k8s-version-262689 kubelet[657]: E0725 19:31:13.215172 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.235906 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:14 old-k8s-version-262689 kubelet[657]: E0725 19:31:14.214851 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.236251 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:26 old-k8s-version-262689 kubelet[657]: E0725 19:31:26.215358 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.236454 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:27 old-k8s-version-262689 kubelet[657]: E0725 19:31:27.215041 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.236868 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:37 old-k8s-version-262689 kubelet[657]: E0725 19:31:37.215248 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.240631 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:40 old-k8s-version-262689 kubelet[657]: E0725 19:31:40.223791 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
W0725 19:34:20.241195 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:52 old-k8s-version-262689 kubelet[657]: E0725 19:31:52.217037 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 1m20s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.241446 688921 logs.go:138] Found kubelet problem: Jul 25 19:31:54 old-k8s-version-262689 kubelet[657]: E0725 19:31:54.225403 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.241652 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:05 old-k8s-version-262689 kubelet[657]: E0725 19:32:05.215333 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.242309 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:07 old-k8s-version-262689 kubelet[657]: E0725 19:32:07.888105 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.242659 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:16 old-k8s-version-262689 kubelet[657]: E0725 19:32:16.271463 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.242899 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:19 old-k8s-version-262689 kubelet[657]: E0725 19:32:19.215018 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.243284 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:28 old-k8s-version-262689 kubelet[657]: E0725 19:32:28.214662 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.243635 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:31 old-k8s-version-262689 kubelet[657]: E0725 19:32:31.215144 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.244269 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:41 old-k8s-version-262689 kubelet[657]: E0725 19:32:41.214882 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.244545 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:46 old-k8s-version-262689 kubelet[657]: E0725 19:32:46.215198 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.244932 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:52 old-k8s-version-262689 kubelet[657]: E0725 19:32:52.215208 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.245138 688921 logs.go:138] Found kubelet problem: Jul 25 19:32:59 old-k8s-version-262689 kubelet[657]: E0725 19:32:59.215146 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.245498 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:06 old-k8s-version-262689 kubelet[657]: E0725 19:33:06.214773 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.245703 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:12 old-k8s-version-262689 kubelet[657]: E0725 19:33:12.219911 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.246101 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:17 old-k8s-version-262689 kubelet[657]: E0725 19:33:17.215219 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.246327 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:25 old-k8s-version-262689 kubelet[657]: E0725 19:33:25.215021 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.246766 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:32 old-k8s-version-262689 kubelet[657]: E0725 19:33:32.216211 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.247012 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:38 old-k8s-version-262689 kubelet[657]: E0725 19:33:38.215054 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.247382 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:45 old-k8s-version-262689 kubelet[657]: E0725 19:33:45.216141 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.247585 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:50 old-k8s-version-262689 kubelet[657]: E0725 19:33:50.215161 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.247936 688921 logs.go:138] Found kubelet problem: Jul 25 19:33:58 old-k8s-version-262689 kubelet[657]: E0725 19:33:58.214660 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.248172 688921 logs.go:138] Found kubelet problem: Jul 25 19:34:03 old-k8s-version-262689 kubelet[657]: E0725 19:34:03.215098 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.248568 688921 logs.go:138] Found kubelet problem: Jul 25 19:34:11 old-k8s-version-262689 kubelet[657]: E0725 19:34:11.214739 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.248773 688921 logs.go:138] Found kubelet problem: Jul 25 19:34:14 old-k8s-version-262689 kubelet[657]: E0725 19:34:14.215104 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
I0725 19:34:20.248804 688921 logs.go:123] Gathering logs for describe nodes ...
I0725 19:34:20.248838 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.20.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
I0725 19:34:20.394413 688921 logs.go:123] Gathering logs for kube-apiserver [8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7] ...
I0725 19:34:20.394445 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7"
I0725 19:34:20.454291 688921 logs.go:123] Gathering logs for kube-apiserver [a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8] ...
I0725 19:34:20.454325 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8"
I0725 19:34:20.507516 688921 logs.go:123] Gathering logs for coredns [6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8] ...
I0725 19:34:20.507595 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8"
I0725 19:34:20.549222 688921 logs.go:123] Gathering logs for kube-scheduler [fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1] ...
I0725 19:34:20.549250 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1"
I0725 19:34:20.600314 688921 logs.go:123] Gathering logs for etcd [bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6] ...
I0725 19:34:20.600352 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6"
I0725 19:34:20.657965 688921 logs.go:123] Gathering logs for kube-proxy [5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f] ...
I0725 19:34:20.657998 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f"
I0725 19:34:20.702882 688921 logs.go:123] Gathering logs for kube-controller-manager [092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb] ...
I0725 19:34:20.702913 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb"
I0725 19:34:20.766370 688921 logs.go:123] Gathering logs for dmesg ...
I0725 19:34:20.766446 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0725 19:34:20.794437 688921 logs.go:123] Gathering logs for coredns [aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05] ...
I0725 19:34:20.794537 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05"
I0725 19:34:20.838887 688921 logs.go:123] Gathering logs for kube-scheduler [2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34] ...
I0725 19:34:20.839007 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34"
I0725 19:34:20.898415 688921 logs.go:123] Gathering logs for storage-provisioner [6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985] ...
I0725 19:34:20.898447 688921 ssh_runner.go:195] Run: /bin/bash -c "sudo /usr/bin/crictl logs --tail 400 6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985"
I0725 19:34:20.947486 688921 out.go:304] Setting ErrFile to fd 2...
I0725 19:34:20.947512 688921 out.go:338] TERM=,COLORTERM=, which probably does not support color
W0725 19:34:20.947573 688921 out.go:239] X Problems detected in kubelet:
W0725 19:34:20.947582 688921 out.go:239] Jul 25 19:33:50 old-k8s-version-262689 kubelet[657]: E0725 19:33:50.215161 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.947592 688921 out.go:239] Jul 25 19:33:58 old-k8s-version-262689 kubelet[657]: E0725 19:33:58.214660 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.947606 688921 out.go:239] Jul 25 19:34:03 old-k8s-version-262689 kubelet[657]: E0725 19:34:03.215098 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
W0725 19:34:20.947612 688921 out.go:239] Jul 25 19:34:11 old-k8s-version-262689 kubelet[657]: E0725 19:34:11.214739 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
W0725 19:34:20.947623 688921 out.go:239] Jul 25 19:34:14 old-k8s-version-262689 kubelet[657]: E0725 19:34:14.215104 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
I0725 19:34:20.947630 688921 out.go:304] Setting ErrFile to fd 2...
I0725 19:34:20.947635 688921 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0725 19:34:23.013238 698181 pod_ready.go:102] pod "coredns-7db6d8ff4d-2dw2z" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:25.016359 698181 pod_ready.go:102] pod "coredns-7db6d8ff4d-2dw2z" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:30.948693 688921 api_server.go:253] Checking apiserver healthz at https://192.168.85.2:8443/healthz ...
I0725 19:34:30.959121 688921 api_server.go:279] https://192.168.85.2:8443/healthz returned 200:
ok
I0725 19:34:30.961497 688921 out.go:177]
W0725 19:34:30.963511 688921 out.go:239] X Exiting due to K8S_UNHEALTHY_CONTROL_PLANE: wait 6m0s for node: wait for healthy API server: controlPlane never updated to v1.20.0
W0725 19:34:30.963612 688921 out.go:239] * Suggestion: Control Plane could not update, try minikube delete --all --purge
W0725 19:34:30.963654 688921 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/11417
W0725 19:34:30.963661 688921 out.go:239] *
W0725 19:34:30.964606 688921 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0725 19:34:30.966662 688921 out.go:177]
I0725 19:34:27.510924 698181 pod_ready.go:102] pod "coredns-7db6d8ff4d-2dw2z" in "kube-system" namespace has status "Ready":"False"
I0725 19:34:29.511238 698181 pod_ready.go:92] pod "coredns-7db6d8ff4d-2dw2z" in "kube-system" namespace has status "Ready":"True"
I0725 19:34:29.511265 698181 pod_ready.go:81] duration metric: took 22.506691607s for pod "coredns-7db6d8ff4d-2dw2z" in "kube-system" namespace to be "Ready" ...
I0725 19:34:29.511278 698181 pod_ready.go:78] waiting up to 6m0s for pod "coredns-7db6d8ff4d-7cfk5" in "kube-system" namespace to be "Ready" ...
I0725 19:34:29.515203 698181 pod_ready.go:97] error getting pod "coredns-7db6d8ff4d-7cfk5" in "kube-system" namespace (skipping!): pods "coredns-7db6d8ff4d-7cfk5" not found
I0725 19:34:29.515231 698181 pod_ready.go:81] duration metric: took 3.94614ms for pod "coredns-7db6d8ff4d-7cfk5" in "kube-system" namespace to be "Ready" ...
E0725 19:34:29.515243 698181 pod_ready.go:66] WaitExtra: waitPodCondition: error getting pod "coredns-7db6d8ff4d-7cfk5" in "kube-system" namespace (skipping!): pods "coredns-7db6d8ff4d-7cfk5" not found
I0725 19:34:29.515250 698181 pod_ready.go:78] waiting up to 6m0s for pod "etcd-embed-certs-240166" in "kube-system" namespace to be "Ready" ...
I0725 19:34:29.523254 698181 pod_ready.go:92] pod "etcd-embed-certs-240166" in "kube-system" namespace has status "Ready":"True"
I0725 19:34:29.523283 698181 pod_ready.go:81] duration metric: took 8.024904ms for pod "etcd-embed-certs-240166" in "kube-system" namespace to be "Ready" ...
I0725 19:34:29.523298 698181 pod_ready.go:78] waiting up to 6m0s for pod "kube-apiserver-embed-certs-240166" in "kube-system" namespace to be "Ready" ...
I0725 19:34:29.530149 698181 pod_ready.go:92] pod "kube-apiserver-embed-certs-240166" in "kube-system" namespace has status "Ready":"True"
I0725 19:34:29.530174 698181 pod_ready.go:81] duration metric: took 6.867689ms for pod "kube-apiserver-embed-certs-240166" in "kube-system" namespace to be "Ready" ...
I0725 19:34:29.530186 698181 pod_ready.go:78] waiting up to 6m0s for pod "kube-controller-manager-embed-certs-240166" in "kube-system" namespace to be "Ready" ...
I0725 19:34:29.536286 698181 pod_ready.go:92] pod "kube-controller-manager-embed-certs-240166" in "kube-system" namespace has status "Ready":"True"
I0725 19:34:29.536321 698181 pod_ready.go:81] duration metric: took 6.11564ms for pod "kube-controller-manager-embed-certs-240166" in "kube-system" namespace to be "Ready" ...
I0725 19:34:29.536334 698181 pod_ready.go:78] waiting up to 6m0s for pod "kube-proxy-xhbp5" in "kube-system" namespace to be "Ready" ...
I0725 19:34:29.708404 698181 pod_ready.go:92] pod "kube-proxy-xhbp5" in "kube-system" namespace has status "Ready":"True"
I0725 19:34:29.708429 698181 pod_ready.go:81] duration metric: took 172.087683ms for pod "kube-proxy-xhbp5" in "kube-system" namespace to be "Ready" ...
I0725 19:34:29.708441 698181 pod_ready.go:78] waiting up to 6m0s for pod "kube-scheduler-embed-certs-240166" in "kube-system" namespace to be "Ready" ...
I0725 19:34:30.111665 698181 pod_ready.go:92] pod "kube-scheduler-embed-certs-240166" in "kube-system" namespace has status "Ready":"True"
I0725 19:34:30.111701 698181 pod_ready.go:81] duration metric: took 403.251052ms for pod "kube-scheduler-embed-certs-240166" in "kube-system" namespace to be "Ready" ...
I0725 19:34:30.111717 698181 pod_ready.go:38] duration metric: took 23.128275631s for extra waiting for all system-critical and pods with labels [k8s-app=kube-dns component=etcd component=kube-apiserver component=kube-controller-manager k8s-app=kube-proxy component=kube-scheduler] to be "Ready" ...
I0725 19:34:30.111733 698181 api_server.go:52] waiting for apiserver process to appear ...
I0725 19:34:30.111824 698181 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0725 19:34:30.128717 698181 api_server.go:72] duration metric: took 24.692109699s to wait for apiserver process to appear ...
I0725 19:34:30.128746 698181 api_server.go:88] waiting for apiserver healthz status ...
I0725 19:34:30.128769 698181 api_server.go:253] Checking apiserver healthz at https://192.168.76.2:8443/healthz ...
I0725 19:34:30.137513 698181 api_server.go:279] https://192.168.76.2:8443/healthz returned 200:
ok
I0725 19:34:30.139011 698181 api_server.go:141] control plane version: v1.30.3
I0725 19:34:30.139047 698181 api_server.go:131] duration metric: took 10.293101ms to wait for apiserver health ...
I0725 19:34:30.139058 698181 system_pods.go:43] waiting for kube-system pods to appear ...
I0725 19:34:30.312461 698181 system_pods.go:59] 8 kube-system pods found
I0725 19:34:30.312498 698181 system_pods.go:61] "coredns-7db6d8ff4d-2dw2z" [4e91d1a1-f51d-4bc9-a312-ab4e94121022] Running
I0725 19:34:30.312504 698181 system_pods.go:61] "etcd-embed-certs-240166" [4c90e835-3da3-4c08-8e1b-fa3c9f8960a3] Running
I0725 19:34:30.312513 698181 system_pods.go:61] "kindnet-7lfwb" [20820be7-3917-4f5c-a617-8f62a5ce1a98] Running
I0725 19:34:30.312517 698181 system_pods.go:61] "kube-apiserver-embed-certs-240166" [189c8487-f49e-40aa-a380-bd16a45294a4] Running
I0725 19:34:30.312522 698181 system_pods.go:61] "kube-controller-manager-embed-certs-240166" [d15a5860-6d3b-41db-8952-42d727fafae2] Running
I0725 19:34:30.312526 698181 system_pods.go:61] "kube-proxy-xhbp5" [e21b820d-585f-4afc-9c1e-25b50d72693c] Running
I0725 19:34:30.312568 698181 system_pods.go:61] "kube-scheduler-embed-certs-240166" [9c92f69c-f335-4f1b-b312-e867ddd98272] Running
I0725 19:34:30.312582 698181 system_pods.go:61] "storage-provisioner" [0b38db65-0f67-4545-8b9c-3272fb58f327] Running
I0725 19:34:30.312587 698181 system_pods.go:74] duration metric: took 173.52404ms to wait for pod list to return data ...
I0725 19:34:30.312595 698181 default_sa.go:34] waiting for default service account to be created ...
I0725 19:34:30.508093 698181 default_sa.go:45] found service account: "default"
I0725 19:34:30.508122 698181 default_sa.go:55] duration metric: took 195.509931ms for default service account to be created ...
I0725 19:34:30.508133 698181 system_pods.go:116] waiting for k8s-apps to be running ...
I0725 19:34:30.712727 698181 system_pods.go:86] 8 kube-system pods found
I0725 19:34:30.712759 698181 system_pods.go:89] "coredns-7db6d8ff4d-2dw2z" [4e91d1a1-f51d-4bc9-a312-ab4e94121022] Running
I0725 19:34:30.712766 698181 system_pods.go:89] "etcd-embed-certs-240166" [4c90e835-3da3-4c08-8e1b-fa3c9f8960a3] Running
I0725 19:34:30.712770 698181 system_pods.go:89] "kindnet-7lfwb" [20820be7-3917-4f5c-a617-8f62a5ce1a98] Running
I0725 19:34:30.712775 698181 system_pods.go:89] "kube-apiserver-embed-certs-240166" [189c8487-f49e-40aa-a380-bd16a45294a4] Running
I0725 19:34:30.712780 698181 system_pods.go:89] "kube-controller-manager-embed-certs-240166" [d15a5860-6d3b-41db-8952-42d727fafae2] Running
I0725 19:34:30.712821 698181 system_pods.go:89] "kube-proxy-xhbp5" [e21b820d-585f-4afc-9c1e-25b50d72693c] Running
I0725 19:34:30.712831 698181 system_pods.go:89] "kube-scheduler-embed-certs-240166" [9c92f69c-f335-4f1b-b312-e867ddd98272] Running
I0725 19:34:30.712835 698181 system_pods.go:89] "storage-provisioner" [0b38db65-0f67-4545-8b9c-3272fb58f327] Running
I0725 19:34:30.712842 698181 system_pods.go:126] duration metric: took 204.703547ms to wait for k8s-apps to be running ...
I0725 19:34:30.712849 698181 system_svc.go:44] waiting for kubelet service to be running ....
I0725 19:34:30.712930 698181 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service kubelet
I0725 19:34:30.725052 698181 system_svc.go:56] duration metric: took 12.19142ms WaitForService to wait for kubelet
I0725 19:34:30.725123 698181 kubeadm.go:582] duration metric: took 25.288521634s to wait for: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0725 19:34:30.725149 698181 node_conditions.go:102] verifying NodePressure condition ...
I0725 19:34:30.908389 698181 node_conditions.go:122] node storage ephemeral capacity is 203034800Ki
I0725 19:34:30.908423 698181 node_conditions.go:123] node cpu capacity is 2
I0725 19:34:30.908441 698181 node_conditions.go:105] duration metric: took 183.284854ms to run NodePressure ...
I0725 19:34:30.908453 698181 start.go:241] waiting for startup goroutines ...
I0725 19:34:30.908461 698181 start.go:246] waiting for cluster config update ...
I0725 19:34:30.908485 698181 start.go:255] writing updated cluster config ...
I0725 19:34:30.908790 698181 ssh_runner.go:195] Run: rm -f paused
I0725 19:34:31.011227 698181 start.go:600] kubectl: 1.30.3, cluster: 1.30.3 (minor skew: 0)
I0725 19:34:31.018101 698181 out.go:177] * Done! kubectl is now configured to use "embed-certs-240166" cluster and "default" namespace by default
==> container status <==
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
8bd318dd74db9 523cad1a4df73 2 minutes ago Exited dashboard-metrics-scraper 5 068b7e93eeb68 dashboard-metrics-scraper-8d5bb5db8-49jgz
1be6aaee701fc ba04bb24b9575 5 minutes ago Running storage-provisioner 2 96d7f668e0fa5 storage-provisioner
22dd933593790 20b332c9a70d8 5 minutes ago Running kubernetes-dashboard 0 79b31dd96a47a kubernetes-dashboard-cd95d586-nkqkw
09fa596a52d12 25a5233254979 5 minutes ago Running kube-proxy 1 e059235025f64 kube-proxy-srbcv
b62507cdbdc5e 1611cd07b61d5 5 minutes ago Running busybox 1 e56250afac817 busybox
aa97a928188d1 db91994f4ee8f 5 minutes ago Running coredns 1 035172f4b1f12 coredns-74ff55c5b-djgf4
6ce89219b0ac0 ba04bb24b9575 5 minutes ago Exited storage-provisioner 1 96d7f668e0fa5 storage-provisioner
ff3fdfb5b2d79 f42786f8afd22 5 minutes ago Running kindnet-cni 1 c0187f0a2471d kindnet-dmcbc
fac92b7ae7a4b e7605f88f17d6 5 minutes ago Running kube-scheduler 1 c2165b2a09c79 kube-scheduler-old-k8s-version-262689
1cd92cb68a1c5 05b738aa1bc63 5 minutes ago Running etcd 1 aa9b38650c1c0 etcd-old-k8s-version-262689
83992657e3b87 1df8a2b116bd1 5 minutes ago Running kube-controller-manager 1 abdada99bd82c kube-controller-manager-old-k8s-version-262689
8282e5e19263c 2c08bbbc02d3a 5 minutes ago Running kube-apiserver 1 17bdf93d666de kube-apiserver-old-k8s-version-262689
6fe11bcfac9b6 1611cd07b61d5 6 minutes ago Exited busybox 0 494de20f4296e busybox
6bec8a2f6c401 db91994f4ee8f 7 minutes ago Exited coredns 0 89f9059ba5055 coredns-74ff55c5b-djgf4
687cf4fb5ec2d f42786f8afd22 7 minutes ago Exited kindnet-cni 0 1153ee1398057 kindnet-dmcbc
5a8e2f746885e 25a5233254979 7 minutes ago Exited kube-proxy 0 ee0dfef3f2c09 kube-proxy-srbcv
2a00adbe91bc4 e7605f88f17d6 8 minutes ago Exited kube-scheduler 0 2e9005a787b56 kube-scheduler-old-k8s-version-262689
092cb7f61fd7d 1df8a2b116bd1 8 minutes ago Exited kube-controller-manager 0 cef425aa8e167 kube-controller-manager-old-k8s-version-262689
a78a4cf991c24 2c08bbbc02d3a 8 minutes ago Exited kube-apiserver 0 45261c58db55a kube-apiserver-old-k8s-version-262689
bbbb23c4486cc 05b738aa1bc63 8 minutes ago Exited etcd 0 2e19fa398b988 etcd-old-k8s-version-262689
==> containerd <==
Jul 25 19:30:40 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:30:40.257524898Z" level=info msg="CreateContainer within sandbox \"068b7e93eeb68f96cac03b6f43d20598972a26327defc6a948bea085e8f1dea2\" for name:\"dashboard-metrics-scraper\" attempt:4 returns container id \"5f99d2a14b7c6e38618baa7187b8c9656dd0546f570fc09eada072d40c5a66b7\""
Jul 25 19:30:40 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:30:40.259593042Z" level=info msg="StartContainer for \"5f99d2a14b7c6e38618baa7187b8c9656dd0546f570fc09eada072d40c5a66b7\""
Jul 25 19:30:40 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:30:40.323289796Z" level=info msg="StartContainer for \"5f99d2a14b7c6e38618baa7187b8c9656dd0546f570fc09eada072d40c5a66b7\" returns successfully"
Jul 25 19:30:40 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:30:40.351197732Z" level=info msg="shim disconnected" id=5f99d2a14b7c6e38618baa7187b8c9656dd0546f570fc09eada072d40c5a66b7 namespace=k8s.io
Jul 25 19:30:40 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:30:40.351269993Z" level=warning msg="cleaning up after shim disconnected" id=5f99d2a14b7c6e38618baa7187b8c9656dd0546f570fc09eada072d40c5a66b7 namespace=k8s.io
Jul 25 19:30:40 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:30:40.351281472Z" level=info msg="cleaning up dead shim" namespace=k8s.io
Jul 25 19:30:40 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:30:40.655096777Z" level=info msg="RemoveContainer for \"80bacb8fee6efe2ae8e2229b095cb84de4c3eb7d0d7c4d07e5488c9a7cd7a8de\""
Jul 25 19:30:40 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:30:40.667884037Z" level=info msg="RemoveContainer for \"80bacb8fee6efe2ae8e2229b095cb84de4c3eb7d0d7c4d07e5488c9a7cd7a8de\" returns successfully"
Jul 25 19:31:40 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:31:40.215454593Z" level=info msg="PullImage \"fake.domain/registry.k8s.io/echoserver:1.4\""
Jul 25 19:31:40 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:31:40.221597464Z" level=info msg="trying next host" error="failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host" host=fake.domain
Jul 25 19:31:40 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:31:40.223275784Z" level=error msg="PullImage \"fake.domain/registry.k8s.io/echoserver:1.4\" failed" error="failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
Jul 25 19:31:40 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:31:40.223368532Z" level=info msg="stop pulling image fake.domain/registry.k8s.io/echoserver:1.4: active requests=0, bytes read=0"
Jul 25 19:32:07 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:32:07.217404172Z" level=info msg="CreateContainer within sandbox \"068b7e93eeb68f96cac03b6f43d20598972a26327defc6a948bea085e8f1dea2\" for container name:\"dashboard-metrics-scraper\" attempt:5"
Jul 25 19:32:07 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:32:07.230756343Z" level=info msg="CreateContainer within sandbox \"068b7e93eeb68f96cac03b6f43d20598972a26327defc6a948bea085e8f1dea2\" for name:\"dashboard-metrics-scraper\" attempt:5 returns container id \"8bd318dd74db97a95d70bc0d9930be8040e1c465947f8087a1f41f53a91b5f2b\""
Jul 25 19:32:07 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:32:07.231504813Z" level=info msg="StartContainer for \"8bd318dd74db97a95d70bc0d9930be8040e1c465947f8087a1f41f53a91b5f2b\""
Jul 25 19:32:07 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:32:07.294192393Z" level=info msg="StartContainer for \"8bd318dd74db97a95d70bc0d9930be8040e1c465947f8087a1f41f53a91b5f2b\" returns successfully"
Jul 25 19:32:07 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:32:07.322524109Z" level=info msg="shim disconnected" id=8bd318dd74db97a95d70bc0d9930be8040e1c465947f8087a1f41f53a91b5f2b namespace=k8s.io
Jul 25 19:32:07 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:32:07.322716211Z" level=warning msg="cleaning up after shim disconnected" id=8bd318dd74db97a95d70bc0d9930be8040e1c465947f8087a1f41f53a91b5f2b namespace=k8s.io
Jul 25 19:32:07 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:32:07.322741844Z" level=info msg="cleaning up dead shim" namespace=k8s.io
Jul 25 19:32:07 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:32:07.888831911Z" level=info msg="RemoveContainer for \"5f99d2a14b7c6e38618baa7187b8c9656dd0546f570fc09eada072d40c5a66b7\""
Jul 25 19:32:07 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:32:07.895316264Z" level=info msg="RemoveContainer for \"5f99d2a14b7c6e38618baa7187b8c9656dd0546f570fc09eada072d40c5a66b7\" returns successfully"
Jul 25 19:34:27 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:34:27.215484345Z" level=info msg="PullImage \"fake.domain/registry.k8s.io/echoserver:1.4\""
Jul 25 19:34:27 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:34:27.232799010Z" level=info msg="trying next host" error="failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host" host=fake.domain
Jul 25 19:34:27 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:34:27.234409886Z" level=error msg="PullImage \"fake.domain/registry.k8s.io/echoserver:1.4\" failed" error="failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
Jul 25 19:34:27 old-k8s-version-262689 containerd[568]: time="2024-07-25T19:34:27.234439267Z" level=info msg="stop pulling image fake.domain/registry.k8s.io/echoserver:1.4: active requests=0, bytes read=0"
==> coredns [6bec8a2f6c401528e85ac844c3e4ceeb5eb110b79993a8dbb471973927177fb8] <==
.:53
[INFO] plugin/reload: Running configuration MD5 = 093a0bf1423dd8c4eee62372bb216168
CoreDNS-1.7.0
linux/arm64, go1.14.4, f59c03d
[INFO] 127.0.0.1:34825 - 33803 "HINFO IN 7787524366633514884.1458302932824696074. udp 57 false 512" NXDOMAIN qr,rd,ra 57 0.04454361s
==> coredns [aa97a928188d1ef74702a4a9c3b633fdd11aba77013410428649b6f03a4dce05] <==
I0725 19:29:16.809575 1 trace.go:116] Trace[2019727887]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125 (started: 2024-07-25 19:28:46.80802639 +0000 UTC m=+0.098858703) (total time: 30.001439165s):
Trace[2019727887]: [30.001439165s] [30.001439165s] END
E0725 19:29:16.809607 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
I0725 19:29:16.809618 1 trace.go:116] Trace[939984059]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125 (started: 2024-07-25 19:28:46.798678329 +0000 UTC m=+0.089510642) (total time: 30.010918948s):
Trace[939984059]: [30.010918948s] [30.010918948s] END
I0725 19:29:16.809627 1 trace.go:116] Trace[1427131847]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125 (started: 2024-07-25 19:28:46.808503383 +0000 UTC m=+0.099335688) (total time: 30.001020386s):
Trace[1427131847]: [30.001020386s] [30.001020386s] END
E0725 19:29:16.809632 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Endpoints: Get "https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
E0725 19:29:16.809631 1 reflector.go:178] pkg/mod/k8s.io/client-go@v0.18.3/tools/cache/reflector.go:125: Failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.96.0.1:443: i/o timeout
.:53
[INFO] plugin/reload: Running configuration MD5 = 093a0bf1423dd8c4eee62372bb216168
CoreDNS-1.7.0
linux/arm64, go1.14.4, f59c03d
[INFO] 127.0.0.1:56372 - 1249 "HINFO IN 5180671977042481159.5930065595496907180. udp 57 false 512" NXDOMAIN qr,rd,ra 57 0.02055083s
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
==> describe nodes <==
Name: old-k8s-version-262689
Roles: control-plane,master
Labels: beta.kubernetes.io/arch=arm64
beta.kubernetes.io/os=linux
kubernetes.io/arch=arm64
kubernetes.io/hostname=old-k8s-version-262689
kubernetes.io/os=linux
minikube.k8s.io/commit=6f15797740a09e0fb947959f5fd09f2e323bde5a
minikube.k8s.io/name=old-k8s-version-262689
minikube.k8s.io/primary=true
minikube.k8s.io/updated_at=2024_07_25T19_26_19_0700
minikube.k8s.io/version=v1.33.1
node-role.kubernetes.io/control-plane=
node-role.kubernetes.io/master=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Thu, 25 Jul 2024 19:26:15 +0000
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: old-k8s-version-262689
AcquireTime: <unset>
RenewTime: Thu, 25 Jul 2024 19:34:24 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Thu, 25 Jul 2024 19:29:32 +0000 Thu, 25 Jul 2024 19:26:08 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Thu, 25 Jul 2024 19:29:32 +0000 Thu, 25 Jul 2024 19:26:08 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Thu, 25 Jul 2024 19:29:32 +0000 Thu, 25 Jul 2024 19:26:08 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Thu, 25 Jul 2024 19:29:32 +0000 Thu, 25 Jul 2024 19:26:34 +0000 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.168.85.2
Hostname: old-k8s-version-262689
Capacity:
cpu: 2
ephemeral-storage: 203034800Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
hugepages-32Mi: 0
hugepages-64Ki: 0
memory: 8022364Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 203034800Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
hugepages-32Mi: 0
hugepages-64Ki: 0
memory: 8022364Ki
pods: 110
System Info:
Machine ID: eff3ebf4acf646da955e7140c1a15fea
System UUID: b2716ca6-790b-4543-b20c-6ea364ff35c0
Boot ID: 6208173b-d514-4152-b2e9-119a649e8fe8
Kernel Version: 5.15.0-1065-aws
OS Image: Ubuntu 22.04.4 LTS
Operating System: linux
Architecture: arm64
Container Runtime Version: containerd://1.7.19
Kubelet Version: v1.20.0
Kube-Proxy Version: v1.20.0
PodCIDR: 10.244.0.0/24
PodCIDRs: 10.244.0.0/24
Non-terminated Pods: (12 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
default busybox 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 6m37s
kube-system coredns-74ff55c5b-djgf4 100m (5%!)(MISSING) 0 (0%!)(MISSING) 70Mi (0%!)(MISSING) 170Mi (2%!)(MISSING) 7m58s
kube-system etcd-old-k8s-version-262689 100m (5%!)(MISSING) 0 (0%!)(MISSING) 100Mi (1%!)(MISSING) 0 (0%!)(MISSING) 8m5s
kube-system kindnet-dmcbc 100m (5%!)(MISSING) 100m (5%!)(MISSING) 50Mi (0%!)(MISSING) 50Mi (0%!)(MISSING) 7m58s
kube-system kube-apiserver-old-k8s-version-262689 250m (12%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 8m5s
kube-system kube-controller-manager-old-k8s-version-262689 200m (10%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 8m5s
kube-system kube-proxy-srbcv 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 7m58s
kube-system kube-scheduler-old-k8s-version-262689 100m (5%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 8m5s
kube-system metrics-server-9975d5f86-8g9gp 100m (5%!)(MISSING) 0 (0%!)(MISSING) 200Mi (2%!)(MISSING) 0 (0%!)(MISSING) 6m27s
kube-system storage-provisioner 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 7m56s
kubernetes-dashboard dashboard-metrics-scraper-8d5bb5db8-49jgz 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 5m31s
kubernetes-dashboard kubernetes-dashboard-cd95d586-nkqkw 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 5m31s
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 950m (47%!)(MISSING) 100m (5%!)(MISSING)
memory 420Mi (5%!)(MISSING) 220Mi (2%!)(MISSING)
ephemeral-storage 100Mi (0%!)(MISSING) 0 (0%!)(MISSING)
hugepages-1Gi 0 (0%!)(MISSING) 0 (0%!)(MISSING)
hugepages-2Mi 0 (0%!)(MISSING) 0 (0%!)(MISSING)
hugepages-32Mi 0 (0%!)(MISSING) 0 (0%!)(MISSING)
hugepages-64Ki 0 (0%!)(MISSING) 0 (0%!)(MISSING)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NodeHasSufficientMemory 8m25s (x5 over 8m26s) kubelet Node old-k8s-version-262689 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 8m25s (x5 over 8m26s) kubelet Node old-k8s-version-262689 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 8m25s (x5 over 8m26s) kubelet Node old-k8s-version-262689 status is now: NodeHasSufficientPID
Normal Starting 8m6s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 8m5s kubelet Node old-k8s-version-262689 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 8m5s kubelet Node old-k8s-version-262689 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 8m5s kubelet Node old-k8s-version-262689 status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 8m5s kubelet Updated Node Allocatable limit across pods
Normal NodeReady 7m58s kubelet Node old-k8s-version-262689 status is now: NodeReady
Normal Starting 7m56s kube-proxy Starting kube-proxy.
Normal Starting 6m kubelet Starting kubelet.
Normal NodeHasSufficientMemory 6m (x8 over 6m) kubelet Node old-k8s-version-262689 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 6m (x8 over 6m) kubelet Node old-k8s-version-262689 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 6m (x7 over 6m) kubelet Node old-k8s-version-262689 status is now: NodeHasSufficientPID
Normal NodeAllocatableEnforced 6m kubelet Updated Node Allocatable limit across pods
Normal Starting 5m45s kube-proxy Starting kube-proxy.
==> dmesg <==
[ +0.001040] FS-Cache: O-key=[8] '0f6fed0000000000'
[ +0.000713] FS-Cache: N-cookie c=0000001e [p=00000015 fl=2 nc=0 na=1]
[ +0.000982] FS-Cache: N-cookie d=0000000057ab999f{9p.inode} n=000000003a1fc167
[ +0.001042] FS-Cache: N-key=[8] '0f6fed0000000000'
[ +0.007051] FS-Cache: Duplicate cookie detected
[ +0.000703] FS-Cache: O-cookie c=00000018 [p=00000015 fl=226 nc=0 na=1]
[ +0.000981] FS-Cache: O-cookie d=0000000057ab999f{9p.inode} n=0000000040afcf4d
[ +0.001044] FS-Cache: O-key=[8] '0f6fed0000000000'
[ +0.000728] FS-Cache: N-cookie c=0000001f [p=00000015 fl=2 nc=0 na=1]
[ +0.000936] FS-Cache: N-cookie d=0000000057ab999f{9p.inode} n=000000002aec574f
[ +0.001060] FS-Cache: N-key=[8] '0f6fed0000000000'
[ +2.902733] FS-Cache: Duplicate cookie detected
[ +0.000731] FS-Cache: O-cookie c=00000016 [p=00000015 fl=226 nc=0 na=1]
[ +0.000958] FS-Cache: O-cookie d=0000000057ab999f{9p.inode} n=000000004200c04f
[ +0.001285] FS-Cache: O-key=[8] '0e6fed0000000000'
[ +0.000753] FS-Cache: N-cookie c=00000021 [p=00000015 fl=2 nc=0 na=1]
[ +0.000959] FS-Cache: N-cookie d=0000000057ab999f{9p.inode} n=000000003a1fc167
[ +0.001072] FS-Cache: N-key=[8] '0e6fed0000000000'
[ +0.318613] FS-Cache: Duplicate cookie detected
[ +0.000752] FS-Cache: O-cookie c=0000001b [p=00000015 fl=226 nc=0 na=1]
[ +0.000982] FS-Cache: O-cookie d=0000000057ab999f{9p.inode} n=00000000b27dda33
[ +0.001094] FS-Cache: O-key=[8] '146fed0000000000'
[ +0.000730] FS-Cache: N-cookie c=00000022 [p=00000015 fl=2 nc=0 na=1]
[ +0.000942] FS-Cache: N-cookie d=0000000057ab999f{9p.inode} n=000000002b202c68
[ +0.001045] FS-Cache: N-key=[8] '146fed0000000000'
==> etcd [1cd92cb68a1c5032b3e67bb3e60e54635f1eab52e0fda097eb7aaf23cbd2b967] <==
2024-07-25 19:30:31.096448 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:30:41.096390 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:30:51.096605 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:31:01.096473 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:31:11.096451 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:31:21.096440 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:31:31.096577 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:31:41.096508 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:31:51.096541 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:32:01.098108 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:32:11.096730 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:32:21.096444 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:32:31.096575 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:32:41.096373 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:32:51.103888 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:33:01.096509 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:33:11.096380 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:33:21.096625 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:33:31.096572 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:33:41.096423 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:33:51.096575 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:34:01.096408 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:34:11.096463 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:34:21.096722 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:34:31.096460 I | etcdserver/api/etcdhttp: /health OK (status code 200)
==> etcd [bbbb23c4486cc3d21d25ad34daca3cfc35fda6d80f6a0e9fcd87f77a79652bb6] <==
2024-07-25 19:26:07.779446 I | embed: listening for peers on 192.168.85.2:2380
2024-07-25 19:26:07.779689 I | embed: listening for metrics on http://127.0.0.1:2381
raft2024/07/25 19:26:08 INFO: 9f0758e1c58a86ed is starting a new election at term 1
raft2024/07/25 19:26:08 INFO: 9f0758e1c58a86ed became candidate at term 2
raft2024/07/25 19:26:08 INFO: 9f0758e1c58a86ed received MsgVoteResp from 9f0758e1c58a86ed at term 2
raft2024/07/25 19:26:08 INFO: 9f0758e1c58a86ed became leader at term 2
raft2024/07/25 19:26:08 INFO: raft.node: 9f0758e1c58a86ed elected leader 9f0758e1c58a86ed at term 2
2024-07-25 19:26:08.460485 I | etcdserver: published {Name:old-k8s-version-262689 ClientURLs:[https://192.168.85.2:2379]} to cluster 68eaea490fab4e05
2024-07-25 19:26:08.460753 I | embed: ready to serve client requests
2024-07-25 19:26:08.462504 I | embed: serving client requests on 127.0.0.1:2379
2024-07-25 19:26:08.462769 I | embed: ready to serve client requests
2024-07-25 19:26:08.463017 I | etcdserver: setting up the initial cluster version to 3.4
2024-07-25 19:26:08.469146 N | etcdserver/membership: set the initial cluster version to 3.4
2024-07-25 19:26:08.473547 I | etcdserver/api: enabled capabilities for version 3.4
2024-07-25 19:26:08.487717 I | embed: serving client requests on 192.168.85.2:2379
2024-07-25 19:26:30.854893 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:26:40.710354 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:26:50.710307 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:27:00.710776 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:27:10.710877 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:27:20.710533 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:27:30.710281 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:27:40.710366 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:27:50.710363 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-07-25 19:28:00.710406 I | etcdserver/api/etcdhttp: /health OK (status code 200)
==> kernel <==
19:34:33 up 3:17, 0 users, load average: 2.28, 2.31, 2.86
Linux old-k8s-version-262689 5.15.0-1065-aws #71~20.04.1-Ubuntu SMP Fri Jun 28 19:59:49 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
PRETTY_NAME="Ubuntu 22.04.4 LTS"
==> kindnet [687cf4fb5ec2da066d588dd1c4d731c898dfd63180c14127e4f87575f4f860a8] <==
I0725 19:26:58.792568 1 main.go:299] handling current node
I0725 19:27:08.792763 1 main.go:295] Handling node with IPs: map[192.168.85.2:{}]
I0725 19:27:08.792973 1 main.go:299] handling current node
W0725 19:27:11.636804 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "namespaces" in API group "" at the cluster scope
E0725 19:27:11.636844 1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "namespaces" in API group "" at the cluster scope
W0725 19:27:12.480906 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: failed to list *v1.NetworkPolicy: networkpolicies.networking.k8s.io is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "networkpolicies" in API group "networking.k8s.io" at the cluster scope
E0725 19:27:12.481002 1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: Failed to watch *v1.NetworkPolicy: failed to list *v1.NetworkPolicy: networkpolicies.networking.k8s.io is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "networkpolicies" in API group "networking.k8s.io" at the cluster scope
W0725 19:27:13.192195 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "pods" in API group "" at the cluster scope
E0725 19:27:13.192445 1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "pods" in API group "" at the cluster scope
I0725 19:27:18.792235 1 main.go:295] Handling node with IPs: map[192.168.85.2:{}]
I0725 19:27:18.792525 1 main.go:299] handling current node
I0725 19:27:28.791847 1 main.go:295] Handling node with IPs: map[192.168.85.2:{}]
I0725 19:27:28.791885 1 main.go:299] handling current node
I0725 19:27:38.792225 1 main.go:295] Handling node with IPs: map[192.168.85.2:{}]
I0725 19:27:38.792295 1 main.go:299] handling current node
I0725 19:27:48.791819 1 main.go:295] Handling node with IPs: map[192.168.85.2:{}]
I0725 19:27:48.791856 1 main.go:299] handling current node
W0725 19:27:55.060099 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "pods" in API group "" at the cluster scope
E0725 19:27:55.060136 1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "pods" in API group "" at the cluster scope
W0725 19:27:56.545757 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "namespaces" in API group "" at the cluster scope
E0725 19:27:56.545803 1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "namespaces" in API group "" at the cluster scope
I0725 19:27:58.792142 1 main.go:295] Handling node with IPs: map[192.168.85.2:{}]
I0725 19:27:58.792247 1 main.go:299] handling current node
W0725 19:28:01.451858 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: failed to list *v1.NetworkPolicy: networkpolicies.networking.k8s.io is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "networkpolicies" in API group "networking.k8s.io" at the cluster scope
E0725 19:28:01.451903 1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: Failed to watch *v1.NetworkPolicy: failed to list *v1.NetworkPolicy: networkpolicies.networking.k8s.io is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "networkpolicies" in API group "networking.k8s.io" at the cluster scope
==> kindnet [ff3fdfb5b2d793d52d00f84a970f452b5e38af8bf34c2152f15e70edaf6cfe01] <==
I0725 19:33:16.708779 1 main.go:299] handling current node
I0725 19:33:26.708442 1 main.go:295] Handling node with IPs: map[192.168.85.2:{}]
I0725 19:33:26.708652 1 main.go:299] handling current node
W0725 19:33:28.342099 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "pods" in API group "" at the cluster scope
E0725 19:33:28.342354 1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "pods" in API group "" at the cluster scope
I0725 19:33:36.708751 1 main.go:295] Handling node with IPs: map[192.168.85.2:{}]
I0725 19:33:36.708849 1 main.go:299] handling current node
W0725 19:33:38.956208 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "namespaces" in API group "" at the cluster scope
E0725 19:33:38.956479 1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "namespaces" in API group "" at the cluster scope
I0725 19:33:46.708682 1 main.go:295] Handling node with IPs: map[192.168.85.2:{}]
I0725 19:33:46.708808 1 main.go:299] handling current node
I0725 19:33:56.708436 1 main.go:295] Handling node with IPs: map[192.168.85.2:{}]
I0725 19:33:56.708474 1 main.go:299] handling current node
I0725 19:34:06.708252 1 main.go:295] Handling node with IPs: map[192.168.85.2:{}]
I0725 19:34:06.708486 1 main.go:299] handling current node
W0725 19:34:08.489871 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: failed to list *v1.NetworkPolicy: networkpolicies.networking.k8s.io is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "networkpolicies" in API group "networking.k8s.io" at the cluster scope
E0725 19:34:08.489919 1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: Failed to watch *v1.NetworkPolicy: failed to list *v1.NetworkPolicy: networkpolicies.networking.k8s.io is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "networkpolicies" in API group "networking.k8s.io" at the cluster scope
I0725 19:34:16.708303 1 main.go:295] Handling node with IPs: map[192.168.85.2:{}]
I0725 19:34:16.708342 1 main.go:299] handling current node
W0725 19:34:23.620509 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "pods" in API group "" at the cluster scope
E0725 19:34:23.620550 1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "pods" in API group "" at the cluster scope
I0725 19:34:26.708718 1 main.go:295] Handling node with IPs: map[192.168.85.2:{}]
I0725 19:34:26.708758 1 main.go:299] handling current node
W0725 19:34:27.652281 1 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "namespaces" in API group "" at the cluster scope
E0725 19:34:27.652336 1 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:kube-system:kindnet" cannot list resource "namespaces" in API group "" at the cluster scope
==> kube-apiserver [8282e5e19263ca28a4b44205c5d5f5dfe8cc3b17c20f817f1123667ae629fca7] <==
I0725 19:31:32.526292 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0725 19:31:32.526301 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
W0725 19:31:46.560434 1 handler_proxy.go:102] no RequestInfo found in the context
E0725 19:31:46.560514 1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0725 19:31:46.560529 1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0725 19:32:08.128885 1 client.go:360] parsed scheme: "passthrough"
I0725 19:32:08.128929 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0725 19:32:08.128937 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0725 19:32:42.457734 1 client.go:360] parsed scheme: "passthrough"
I0725 19:32:42.457794 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0725 19:32:42.457803 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0725 19:33:17.747452 1 client.go:360] parsed scheme: "passthrough"
I0725 19:33:17.747504 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0725 19:33:17.747514 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
W0725 19:33:42.871104 1 handler_proxy.go:102] no RequestInfo found in the context
E0725 19:33:42.871468 1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0725 19:33:42.871630 1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0725 19:33:51.746061 1 client.go:360] parsed scheme: "passthrough"
I0725 19:33:51.746158 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0725 19:33:51.746192 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0725 19:34:21.860139 1 client.go:360] parsed scheme: "passthrough"
I0725 19:34:21.860187 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0725 19:34:21.860196 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
==> kube-apiserver [a78a4cf991c24677e1aee0c57c3053ec9d18ed4f3b954e0b55dbaeb5bc2cd3e8] <==
I0725 19:26:16.076223 1 controller.go:132] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
I0725 19:26:16.076259 1 controller.go:132] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
I0725 19:26:16.084684 1 storage_scheduling.go:132] created PriorityClass system-node-critical with value 2000001000
I0725 19:26:16.089881 1 storage_scheduling.go:132] created PriorityClass system-cluster-critical with value 2000000000
I0725 19:26:16.090009 1 storage_scheduling.go:148] all system priority classes are created successfully or already exist.
I0725 19:26:16.613495 1 controller.go:606] quota admission added evaluator for: roles.rbac.authorization.k8s.io
I0725 19:26:16.660001 1 controller.go:606] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
W0725 19:26:16.813366 1 lease.go:233] Resetting endpoints for master service "kubernetes" to [192.168.85.2]
I0725 19:26:16.814977 1 controller.go:606] quota admission added evaluator for: endpoints
I0725 19:26:16.825164 1 controller.go:606] quota admission added evaluator for: endpointslices.discovery.k8s.io
I0725 19:26:17.732041 1 controller.go:606] quota admission added evaluator for: serviceaccounts
I0725 19:26:18.533401 1 controller.go:606] quota admission added evaluator for: deployments.apps
I0725 19:26:18.611285 1 controller.go:606] quota admission added evaluator for: daemonsets.apps
I0725 19:26:26.984093 1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io
I0725 19:26:34.390642 1 controller.go:606] quota admission added evaluator for: controllerrevisions.apps
I0725 19:26:34.627741 1 controller.go:606] quota admission added evaluator for: replicasets.apps
I0725 19:26:46.162715 1 client.go:360] parsed scheme: "passthrough"
I0725 19:26:46.163005 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0725 19:26:46.163059 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0725 19:27:23.306720 1 client.go:360] parsed scheme: "passthrough"
I0725 19:27:23.306777 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0725 19:27:23.306787 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
I0725 19:28:03.292393 1 client.go:360] parsed scheme: "passthrough"
I0725 19:28:03.292455 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>}
I0725 19:28:03.292464 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
==> kube-controller-manager [092cb7f61fd7d0ee33ca01ee05c0526235ab4d055f8a55aebbdc0609b6c057bb] <==
I0725 19:26:34.569819 1 shared_informer.go:247] Caches are synced for certificate-csrsigning-kubelet-client
I0725 19:26:34.570012 1 shared_informer.go:247] Caches are synced for certificate-csrsigning-kube-apiserver-client
I0725 19:26:34.573743 1 shared_informer.go:247] Caches are synced for certificate-csrapproving
I0725 19:26:34.581475 1 shared_informer.go:247] Caches are synced for resource quota
I0725 19:26:34.594967 1 shared_informer.go:247] Caches are synced for PVC protection
I0725 19:26:34.624641 1 shared_informer.go:247] Caches are synced for expand
I0725 19:26:34.624641 1 shared_informer.go:247] Caches are synced for stateful set
I0725 19:26:34.627689 1 shared_informer.go:247] Caches are synced for persistent volume
I0725 19:26:34.629207 1 shared_informer.go:247] Caches are synced for resource quota
I0725 19:26:34.651266 1 shared_informer.go:247] Caches are synced for attach detach
I0725 19:26:34.662277 1 event.go:291] "Event occurred" object="kube-system/coredns" kind="Deployment" apiVersion="apps/v1" type="Normal" reason="ScalingReplicaSet" message="Scaled up replica set coredns-74ff55c5b to 2"
I0725 19:26:34.743829 1 shared_informer.go:240] Waiting for caches to sync for garbage collector
I0725 19:26:34.767820 1 event.go:291] "Event occurred" object="kube-system/coredns-74ff55c5b" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: coredns-74ff55c5b-dcw28"
E0725 19:26:34.801086 1 daemon_controller.go:320] kube-system/kindnet failed with : error storing status for daemon set &v1.DaemonSet{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kindnet", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"e597a14f-be32-4fd9-a4b1-85b8ad31516e", ResourceVersion:"281", Generation:1, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63857532379, loc:(*time.Location)(0x632eb80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"kindnet", "k8s-app":"kindnet", "tier":"node"}, Annotations:map[string]string{"deprecated.daemonset.template.generation":"1", "kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"DaemonSet\",\"metadata\":{\"annotations\":{},\"labels\":{\"app\":\"kindnet\",\"k8s-app\":\"kindnet\",\"tier\":\"node\"},\"name\":\"kindnet\",\"namespace\":\"kube-system\"},\"spec\":{\"selector\":{\"matchLabels\":{\"app\":\"k
indnet\"}},\"template\":{\"metadata\":{\"labels\":{\"app\":\"kindnet\",\"k8s-app\":\"kindnet\",\"tier\":\"node\"}},\"spec\":{\"containers\":[{\"env\":[{\"name\":\"HOST_IP\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.hostIP\"}}},{\"name\":\"POD_IP\",\"valueFrom\":{\"fieldRef\":{\"fieldPath\":\"status.podIP\"}}},{\"name\":\"POD_SUBNET\",\"value\":\"10.244.0.0/16\"}],\"image\":\"docker.io/kindest/kindnetd:v20240719-e7903573\",\"name\":\"kindnet-cni\",\"resources\":{\"limits\":{\"cpu\":\"100m\",\"memory\":\"50Mi\"},\"requests\":{\"cpu\":\"100m\",\"memory\":\"50Mi\"}},\"securityContext\":{\"capabilities\":{\"add\":[\"NET_RAW\",\"NET_ADMIN\"]},\"privileged\":false},\"volumeMounts\":[{\"mountPath\":\"/etc/cni/net.d\",\"name\":\"cni-cfg\"},{\"mountPath\":\"/run/xtables.lock\",\"name\":\"xtables-lock\",\"readOnly\":false},{\"mountPath\":\"/lib/modules\",\"name\":\"lib-modules\",\"readOnly\":true}]}],\"hostNetwork\":true,\"serviceAccountName\":\"kindnet\",\"tolerations\":[{\"effect\":\"NoSchedule\",\"operator\
":\"Exists\"}],\"volumes\":[{\"hostPath\":{\"path\":\"/etc/cni/net.d\",\"type\":\"DirectoryOrCreate\"},\"name\":\"cni-cfg\"},{\"hostPath\":{\"path\":\"/run/xtables.lock\",\"type\":\"FileOrCreate\"},\"name\":\"xtables-lock\"},{\"hostPath\":{\"path\":\"/lib/modules\"},\"name\":\"lib-modules\"}]}}}}\n"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kubectl-client-side-apply", Operation:"Update", APIVersion:"apps/v1", Time:(*v1.Time)(0x40012af880), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0x40012af8a0)}}}, Spec:v1.DaemonSetSpec{Selector:(*v1.LabelSelector)(0x40012af8c0), Template:v1.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string
{"app":"kindnet", "k8s-app":"kindnet", "tier":"node"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"cni-cfg", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0x40012af8e0), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil),
FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil), Ephemeral:(*v1.EphemeralVolumeSource)(nil)}}, v1.Volume{Name:"xtables-lock", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0x40012af900), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.Glust
erfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil), Ephemeral:(*v1.EphemeralVolumeSource)(nil)}}, v1.Volume{Name:"lib-modules", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(0x40012af920), EmptyDi
r:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(nil), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil),
PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil), Ephemeral:(*v1.EphemeralVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"kindnet-cni", Image:"docker.io/kindest/kindnetd:v20240719-e7903573", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort(nil), EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar{v1.EnvVar{Name:"HOST_IP", Value:"", ValueFrom:(*v1.EnvVarSource)(0x40012af940)}, v1.EnvVar{Name:"POD_IP", Value:"", ValueFrom:(*v1.EnvVarSource)(0x40012af980)}, v1.EnvVar{Name:"POD_SUBNET", Value:"10.244.0.0/16", ValueFrom:(*v1.EnvVarSource)(nil)}}, Resources:v1.ResourceRequirements{Limits:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:100, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:52428800, scale:
0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"50Mi", Format:"BinarySI"}}, Requests:v1.ResourceList{"cpu":resource.Quantity{i:resource.int64Amount{value:100, scale:-3}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100m", Format:"DecimalSI"}, "memory":resource.Quantity{i:resource.int64Amount{value:52428800, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"50Mi", Format:"BinarySI"}}}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"cni-cfg", ReadOnly:false, MountPath:"/etc/cni/net.d", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}, v1.VolumeMount{Name:"xtables-lock", ReadOnly:false, MountPath:"/run/xtables.lock", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}, v1.VolumeMount{Name:"lib-modules", ReadOnly:true, MountPath:"/lib/modules", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), StartupProbe:
(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"IfNotPresent", SecurityContext:(*v1.SecurityContext)(0x400111c5a0), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]v1.EphemeralContainer(nil), RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0x40007153c8), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"kindnet", DeprecatedServiceAccount:"kindnet", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", HostNetwork:true, HostPID:false, HostIPC:false, ShareProcessNamespace:(*bool)(nil), SecurityContext:(*v1.PodSecurityContext)(0x4000090ee0), ImagePullSecrets:[]v1.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"", Operator:"Exists", Value:"", Effect:"NoSchedule", TolerationSeconds:(*int64)(nil)}},
HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), DNSConfig:(*v1.PodDNSConfig)(nil), ReadinessGates:[]v1.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), EnableServiceLinks:(*bool)(nil), PreemptionPolicy:(*v1.PreemptionPolicy)(nil), Overhead:v1.ResourceList(nil), TopologySpreadConstraints:[]v1.TopologySpreadConstraint(nil), SetHostnameAsFQDN:(*bool)(nil)}}, UpdateStrategy:v1.DaemonSetUpdateStrategy{Type:"RollingUpdate", RollingUpdate:(*v1.RollingUpdateDaemonSet)(0x4000947f18)}, MinReadySeconds:0, RevisionHistoryLimit:(*int32)(0x4000715420)}, Status:v1.DaemonSetStatus{CurrentNumberScheduled:0, NumberMisscheduled:0, DesiredNumberScheduled:0, NumberReady:0, ObservedGeneration:0, UpdatedNumberScheduled:0, NumberAvailable:0, NumberUnavailable:0, CollisionCount:(*int32)(nil), Conditions:[]v1.DaemonSetCondition(nil)}}: Operation cannot be fulfilled on daemonsets.apps "kindnet": the object has been modified; please apply your changes to the latest version and try again
I0725 19:26:34.812668 1 event.go:291] "Event occurred" object="kube-system/coredns-74ff55c5b" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: coredns-74ff55c5b-djgf4"
I0725 19:26:35.044075 1 shared_informer.go:247] Caches are synced for garbage collector
I0725 19:26:35.074540 1 shared_informer.go:247] Caches are synced for garbage collector
I0725 19:26:35.074560 1 garbagecollector.go:151] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
I0725 19:26:37.259143 1 event.go:291] "Event occurred" object="kube-system/coredns" kind="Deployment" apiVersion="apps/v1" type="Normal" reason="ScalingReplicaSet" message="Scaled down replica set coredns-74ff55c5b to 1"
I0725 19:26:37.280213 1 event.go:291] "Event occurred" object="kube-system/coredns-74ff55c5b" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulDelete" message="Deleted pod: coredns-74ff55c5b-dcw28"
I0725 19:26:39.331772 1 node_lifecycle_controller.go:1222] Controller detected that some Nodes are Ready. Exiting master disruption mode.
I0725 19:28:03.964837 1 event.go:291] "Event occurred" object="kube-system/metrics-server" kind="Deployment" apiVersion="apps/v1" type="Normal" reason="ScalingReplicaSet" message="Scaled up replica set metrics-server-9975d5f86 to 1"
E0725 19:28:04.037807 1 clusterroleaggregation_controller.go:181] edit failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "edit": the object has been modified; please apply your changes to the latest version and try again
E0725 19:28:04.043288 1 clusterroleaggregation_controller.go:181] view failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "view": the object has been modified; please apply your changes to the latest version and try again
E0725 19:28:04.070020 1 clusterroleaggregation_controller.go:181] admin failed with : Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "admin": the object has been modified; please apply your changes to the latest version and try again
==> kube-controller-manager [83992657e3b879a7a80d6d310f516828b9ac2d505791880fb62eabc3fdc281b5] <==
W0725 19:30:09.595896 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0725 19:30:33.248269 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0725 19:30:41.246367 1 request.go:655] Throttling request took 1.048193082s, request: GET:https://192.168.85.2:8443/apis/extensions/v1beta1?timeout=32s
W0725 19:30:42.097913 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0725 19:31:03.749995 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0725 19:31:13.748525 1 request.go:655] Throttling request took 1.046114447s, request: GET:https://192.168.85.2:8443/apis/extensions/v1beta1?timeout=32s
W0725 19:31:14.600065 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0725 19:31:34.251891 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0725 19:31:46.250482 1 request.go:655] Throttling request took 1.04449563s, request: GET:https://192.168.85.2:8443/apis/extensions/v1beta1?timeout=32s
W0725 19:31:47.103438 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0725 19:32:04.753952 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0725 19:32:18.753869 1 request.go:655] Throttling request took 1.048489881s, request: GET:https://192.168.85.2:8443/apis/extensions/v1beta1?timeout=32s
W0725 19:32:19.605454 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0725 19:32:35.255773 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0725 19:32:51.255958 1 request.go:655] Throttling request took 1.048228753s, request: GET:https://192.168.85.2:8443/apis/extensions/v1beta1?timeout=32s
W0725 19:32:52.107796 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0725 19:33:05.757577 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0725 19:33:23.758342 1 request.go:655] Throttling request took 1.046343956s, request: GET:https://192.168.85.2:8443/apis/extensions/v1beta1?timeout=32s
W0725 19:33:24.609973 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0725 19:33:36.259638 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0725 19:33:56.260611 1 request.go:655] Throttling request took 1.048029691s, request: GET:https://192.168.85.2:8443/apis/apiextensions.k8s.io/v1beta1?timeout=32s
W0725 19:33:57.112288 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
E0725 19:34:06.766541 1 resource_quota_controller.go:409] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0725 19:34:28.762776 1 request.go:655] Throttling request took 1.048268639s, request: GET:https://192.168.85.2:8443/apis/apiextensions.k8s.io/v1?timeout=32s
W0725 19:34:29.614286 1 garbagecollector.go:703] failed to discover some groups: map[metrics.k8s.io/v1beta1:the server is currently unable to handle the request]
==> kube-proxy [09fa596a52d128c9994b4384da212aa9349f6711170475aac344eeb7c10c8c98] <==
I0725 19:28:47.039278 1 node.go:172] Successfully retrieved node IP: 192.168.85.2
I0725 19:28:47.039571 1 server_others.go:142] kube-proxy node IP is an IPv4 address (192.168.85.2), assume IPv4 operation
W0725 19:28:47.076559 1 server_others.go:578] Unknown proxy mode "", assuming iptables proxy
I0725 19:28:47.076653 1 server_others.go:185] Using iptables Proxier.
I0725 19:28:47.076990 1 server.go:650] Version: v1.20.0
I0725 19:28:47.098913 1 config.go:315] Starting service config controller
I0725 19:28:47.098993 1 shared_informer.go:240] Waiting for caches to sync for service config
I0725 19:28:47.099082 1 config.go:224] Starting endpoint slice config controller
I0725 19:28:47.099092 1 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
I0725 19:28:47.200189 1 shared_informer.go:247] Caches are synced for endpoint slice config
I0725 19:28:47.200388 1 shared_informer.go:247] Caches are synced for service config
==> kube-proxy [5a8e2f746885e19969520172b856da71d30879fb506af5db848f0731b0f8ee1f] <==
I0725 19:26:36.228977 1 node.go:172] Successfully retrieved node IP: 192.168.85.2
I0725 19:26:36.229068 1 server_others.go:142] kube-proxy node IP is an IPv4 address (192.168.85.2), assume IPv4 operation
W0725 19:26:36.364091 1 server_others.go:578] Unknown proxy mode "", assuming iptables proxy
I0725 19:26:36.364286 1 server_others.go:185] Using iptables Proxier.
I0725 19:26:36.367590 1 server.go:650] Version: v1.20.0
I0725 19:26:36.379296 1 config.go:315] Starting service config controller
I0725 19:26:36.380071 1 config.go:224] Starting endpoint slice config controller
I0725 19:26:36.380958 1 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
I0725 19:26:36.381144 1 shared_informer.go:240] Waiting for caches to sync for service config
I0725 19:26:36.481091 1 shared_informer.go:247] Caches are synced for endpoint slice config
I0725 19:26:36.482549 1 shared_informer.go:247] Caches are synced for service config
==> kube-scheduler [2a00adbe91bc43396fb4655279df11d46cadb9750a439b2afc0bd4ab7294bd34] <==
W0725 19:26:15.245763 1 authentication.go:332] Error looking up in-cluster authentication configuration: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot get resource "configmaps" in API group "" in the namespace "kube-system"
W0725 19:26:15.245776 1 authentication.go:333] Continuing without authentication configuration. This may treat all requests as anonymous.
W0725 19:26:15.245781 1 authentication.go:334] To require authentication configuration lookup to succeed, set --authentication-tolerate-lookup-failure=false
I0725 19:26:15.301844 1 secure_serving.go:197] Serving securely on 127.0.0.1:10259
I0725 19:26:15.302095 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0725 19:26:15.302111 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0725 19:26:15.302248 1 tlsconfig.go:240] Starting DynamicServingCertificateController
E0725 19:26:15.344716 1 reflector.go:138] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
E0725 19:26:15.344716 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0725 19:26:15.345027 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0725 19:26:15.345264 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0725 19:26:15.345519 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0725 19:26:15.347651 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0725 19:26:15.347730 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0725 19:26:15.348446 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0725 19:26:15.348511 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0725 19:26:15.348757 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0725 19:26:15.349339 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0725 19:26:15.351466 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0725 19:26:16.193754 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0725 19:26:16.363249 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0725 19:26:16.417202 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0725 19:26:16.432931 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0725 19:26:16.432937 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
I0725 19:26:16.702189 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
==> kube-scheduler [fac92b7ae7a4bca579e6e42dde322f2efd70be0bd69a7361cbdeceff14266ce1] <==
I0725 19:28:35.037212 1 serving.go:331] Generated self-signed cert in-memory
I0725 19:28:42.962518 1 secure_serving.go:197] Serving securely on 127.0.0.1:10259
I0725 19:28:42.962614 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0725 19:28:42.962620 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0725 19:28:42.962632 1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0725 19:28:42.979037 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0725 19:28:42.979069 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0725 19:28:42.979090 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0725 19:28:42.979094 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0725 19:28:43.063085 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I0725 19:28:43.083457 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0725 19:28:43.083514 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
==> kubelet <==
Jul 25 19:32:59 old-k8s-version-262689 kubelet[657]: E0725 19:32:59.215146 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Jul 25 19:33:06 old-k8s-version-262689 kubelet[657]: I0725 19:33:06.214417 657 scope.go:95] [topologymanager] RemoveContainer - Container ID: 8bd318dd74db97a95d70bc0d9930be8040e1c465947f8087a1f41f53a91b5f2b
Jul 25 19:33:06 old-k8s-version-262689 kubelet[657]: E0725 19:33:06.214773 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
Jul 25 19:33:12 old-k8s-version-262689 kubelet[657]: E0725 19:33:12.219911 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Jul 25 19:33:17 old-k8s-version-262689 kubelet[657]: I0725 19:33:17.214298 657 scope.go:95] [topologymanager] RemoveContainer - Container ID: 8bd318dd74db97a95d70bc0d9930be8040e1c465947f8087a1f41f53a91b5f2b
Jul 25 19:33:17 old-k8s-version-262689 kubelet[657]: E0725 19:33:17.215219 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
Jul 25 19:33:25 old-k8s-version-262689 kubelet[657]: E0725 19:33:25.215021 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Jul 25 19:33:32 old-k8s-version-262689 kubelet[657]: I0725 19:33:32.214810 657 scope.go:95] [topologymanager] RemoveContainer - Container ID: 8bd318dd74db97a95d70bc0d9930be8040e1c465947f8087a1f41f53a91b5f2b
Jul 25 19:33:32 old-k8s-version-262689 kubelet[657]: E0725 19:33:32.216211 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
Jul 25 19:33:38 old-k8s-version-262689 kubelet[657]: E0725 19:33:38.215054 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Jul 25 19:33:45 old-k8s-version-262689 kubelet[657]: I0725 19:33:45.215611 657 scope.go:95] [topologymanager] RemoveContainer - Container ID: 8bd318dd74db97a95d70bc0d9930be8040e1c465947f8087a1f41f53a91b5f2b
Jul 25 19:33:45 old-k8s-version-262689 kubelet[657]: E0725 19:33:45.216141 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
Jul 25 19:33:50 old-k8s-version-262689 kubelet[657]: E0725 19:33:50.215161 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Jul 25 19:33:58 old-k8s-version-262689 kubelet[657]: I0725 19:33:58.214315 657 scope.go:95] [topologymanager] RemoveContainer - Container ID: 8bd318dd74db97a95d70bc0d9930be8040e1c465947f8087a1f41f53a91b5f2b
Jul 25 19:33:58 old-k8s-version-262689 kubelet[657]: E0725 19:33:58.214660 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
Jul 25 19:34:03 old-k8s-version-262689 kubelet[657]: E0725 19:34:03.215098 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Jul 25 19:34:11 old-k8s-version-262689 kubelet[657]: I0725 19:34:11.214384 657 scope.go:95] [topologymanager] RemoveContainer - Container ID: 8bd318dd74db97a95d70bc0d9930be8040e1c465947f8087a1f41f53a91b5f2b
Jul 25 19:34:11 old-k8s-version-262689 kubelet[657]: E0725 19:34:11.214739 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
Jul 25 19:34:14 old-k8s-version-262689 kubelet[657]: E0725 19:34:14.215104 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ImagePullBackOff: "Back-off pulling image \"fake.domain/registry.k8s.io/echoserver:1.4\""
Jul 25 19:34:23 old-k8s-version-262689 kubelet[657]: I0725 19:34:23.214346 657 scope.go:95] [topologymanager] RemoveContainer - Container ID: 8bd318dd74db97a95d70bc0d9930be8040e1c465947f8087a1f41f53a91b5f2b
Jul 25 19:34:23 old-k8s-version-262689 kubelet[657]: E0725 19:34:23.214716 657 pod_workers.go:191] Error syncing pod d35a2687-c50f-4572-81b9-45b07e9c77a7 ("dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"), skipping: failed to "StartContainer" for "dashboard-metrics-scraper" with CrashLoopBackOff: "back-off 2m40s restarting failed container=dashboard-metrics-scraper pod=dashboard-metrics-scraper-8d5bb5db8-49jgz_kubernetes-dashboard(d35a2687-c50f-4572-81b9-45b07e9c77a7)"
Jul 25 19:34:27 old-k8s-version-262689 kubelet[657]: E0725 19:34:27.234738 657 remote_image.go:113] PullImage "fake.domain/registry.k8s.io/echoserver:1.4" from image service failed: rpc error: code = Unknown desc = failed to pull and unpack image "fake.domain/registry.k8s.io/echoserver:1.4": failed to resolve reference "fake.domain/registry.k8s.io/echoserver:1.4": failed to do request: Head "https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host
Jul 25 19:34:27 old-k8s-version-262689 kubelet[657]: E0725 19:34:27.234796 657 kuberuntime_image.go:51] Pull image "fake.domain/registry.k8s.io/echoserver:1.4" failed: rpc error: code = Unknown desc = failed to pull and unpack image "fake.domain/registry.k8s.io/echoserver:1.4": failed to resolve reference "fake.domain/registry.k8s.io/echoserver:1.4": failed to do request: Head "https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host
Jul 25 19:34:27 old-k8s-version-262689 kubelet[657]: E0725 19:34:27.234979 657 kuberuntime_manager.go:829] container &Container{Name:metrics-server,Image:fake.domain/registry.k8s.io/echoserver:1.4,Command:[],Args:[--cert-dir=/tmp --secure-port=4443 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --kubelet-use-node-status-port --metric-resolution=60s --kubelet-insecure-tls],WorkingDir:,Ports:[]ContainerPort{ContainerPort{Name:https,HostPort:0,ContainerPort:4443,Protocol:TCP,HostIP:,},},Env:[]EnvVar{},Resources:ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{100 -3} {<nil>} 100m DecimalSI},memory: {{209715200 0} {<nil>} BinarySI},},},VolumeMounts:[]VolumeMount{VolumeMount{Name:tmp-dir,ReadOnly:false,MountPath:/tmp,SubPath:,MountPropagation:nil,SubPathExpr:,},VolumeMount{Name:metrics-server-token-wbrxz,ReadOnly:true,MountPath:/var/run/secrets/kubernetes.io/serviceaccount,SubPath:,MountPropagation:nil,SubPathExpr:,},},LivenessProbe:&Probe{Handler:Handler{Exec
:nil,HTTPGet:&HTTPGetAction{Path:/livez,Port:{1 0 https},Host:,Scheme:HTTPS,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,},ReadinessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/readyz,Port:{1 0 https},Host:,Scheme:HTTPS,HTTPHeaders:[]HTTPHeader{},},TCPSocket:nil,},InitialDelaySeconds:0,TimeoutSeconds:1,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:3,},Lifecycle:nil,TerminationMessagePath:/dev/termination-log,ImagePullPolicy:IfNotPresent,SecurityContext:&SecurityContext{Capabilities:nil,Privileged:nil,SELinuxOptions:nil,RunAsUser:*1000,RunAsNonRoot:*true,ReadOnlyRootFilesystem:*true,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,WindowsOptions:nil,SeccompProfile:nil,},Stdin:false,StdinOnce:false,TTY:false,EnvFrom:[]EnvFromSource{},TerminationMessagePolicy:File,VolumeDevices:[]VolumeDevice{},StartupProbe:nil,} start failed in pod metrics-server-9975d5f86-8g9gp_kube-system(3897709
f-bdc3-4e64-9947-4512473cf65b): ErrImagePull: rpc error: code = Unknown desc = failed to pull and unpack image "fake.domain/registry.k8s.io/echoserver:1.4": failed to resolve reference "fake.domain/registry.k8s.io/echoserver:1.4": failed to do request: Head "https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host
Jul 25 19:34:27 old-k8s-version-262689 kubelet[657]: E0725 19:34:27.235018 657 pod_workers.go:191] Error syncing pod 3897709f-bdc3-4e64-9947-4512473cf65b ("metrics-server-9975d5f86-8g9gp_kube-system(3897709f-bdc3-4e64-9947-4512473cf65b)"), skipping: failed to "StartContainer" for "metrics-server" with ErrImagePull: "rpc error: code = Unknown desc = failed to pull and unpack image \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to resolve reference \"fake.domain/registry.k8s.io/echoserver:1.4\": failed to do request: Head \"https://fake.domain/v2/registry.k8s.io/echoserver/manifests/1.4\": dial tcp: lookup fake.domain on 192.168.85.1:53: no such host"
==> kubernetes-dashboard [22dd93359379075cdee6cc988a33beecf77064ec8b1c1b61b00788a08495e2cc] <==
2024/07/25 19:29:11 Starting overwatch
2024/07/25 19:29:11 Using namespace: kubernetes-dashboard
2024/07/25 19:29:11 Using in-cluster config to connect to apiserver
2024/07/25 19:29:11 Using secret token for csrf signing
2024/07/25 19:29:11 Initializing csrf token from kubernetes-dashboard-csrf secret
2024/07/25 19:29:11 Empty token. Generating and storing in a secret kubernetes-dashboard-csrf
2024/07/25 19:29:11 Successful initial request to the apiserver, version: v1.20.0
2024/07/25 19:29:11 Generating JWE encryption key
2024/07/25 19:29:11 New synchronizer has been registered: kubernetes-dashboard-key-holder-kubernetes-dashboard. Starting
2024/07/25 19:29:11 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kubernetes-dashboard
2024/07/25 19:29:12 Initializing JWE encryption key from synchronized object
2024/07/25 19:29:12 Creating in-cluster Sidecar client
2024/07/25 19:29:12 Serving insecurely on HTTP port: 9090
2024/07/25 19:29:12 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/07/25 19:29:42 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/07/25 19:30:12 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/07/25 19:30:42 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/07/25 19:31:12 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/07/25 19:31:42 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/07/25 19:32:12 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/07/25 19:32:42 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/07/25 19:33:12 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/07/25 19:33:42 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2024/07/25 19:34:12 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
==> storage-provisioner [1be6aaee701fc8139b79bc32fca06eada56a5b8869fdc79b4a598c2c2505462e] <==
I0725 19:29:32.412267 1 storage_provisioner.go:116] Initializing the minikube storage provisioner...
I0725 19:29:32.425429 1 storage_provisioner.go:141] Storage provisioner initialized, now starting service!
I0725 19:29:32.425663 1 leaderelection.go:243] attempting to acquire leader lease kube-system/k8s.io-minikube-hostpath...
I0725 19:29:49.886192 1 leaderelection.go:253] successfully acquired lease kube-system/k8s.io-minikube-hostpath
I0725 19:29:49.886594 1 controller.go:835] Starting provisioner controller k8s.io/minikube-hostpath_old-k8s-version-262689_debc9e94-1a68-4548-aa12-7d560a98bc49!
I0725 19:29:49.887368 1 event.go:282] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"k8s.io-minikube-hostpath", UID:"9b2358de-70e0-4b4a-ab15-077bee6a92f1", APIVersion:"v1", ResourceVersion:"836", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' old-k8s-version-262689_debc9e94-1a68-4548-aa12-7d560a98bc49 became leader
I0725 19:29:49.986838 1 controller.go:884] Started provisioner controller k8s.io/minikube-hostpath_old-k8s-version-262689_debc9e94-1a68-4548-aa12-7d560a98bc49!
==> storage-provisioner [6ce89219b0ac011ccb222d7354ece78a535ed3974879be78466dbad059580985] <==
I0725 19:28:46.641964 1 storage_provisioner.go:116] Initializing the minikube storage provisioner...
F0725 19:29:16.664119 1 main.go:39] error getting server version: Get "https://10.96.0.1:443/version?timeout=32s": dial tcp 10.96.0.1:443: i/o timeout
-- /stdout --
helpers_test.go:254: (dbg) Run: out/minikube-linux-arm64 status --format={{.APIServer}} -p old-k8s-version-262689 -n old-k8s-version-262689
helpers_test.go:261: (dbg) Run: kubectl --context old-k8s-version-262689 get po -o=jsonpath={.items[*].metadata.name} -A --field-selector=status.phase!=Running
helpers_test.go:272: non-running pods: metrics-server-9975d5f86-8g9gp
helpers_test.go:274: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/SecondStart]: describe non-running pods <======
helpers_test.go:277: (dbg) Run: kubectl --context old-k8s-version-262689 describe pod metrics-server-9975d5f86-8g9gp
helpers_test.go:277: (dbg) Non-zero exit: kubectl --context old-k8s-version-262689 describe pod metrics-server-9975d5f86-8g9gp: exit status 1 (102.536098ms)
** stderr **
Error from server (NotFound): pods "metrics-server-9975d5f86-8g9gp" not found
** /stderr **
helpers_test.go:279: kubectl --context old-k8s-version-262689 describe pod metrics-server-9975d5f86-8g9gp: exit status 1
--- FAIL: TestStartStop/group/old-k8s-version/serial/SecondStart (377.86s)