=== RUN TestKubernetesUpgrade
=== PAUSE TestKubernetesUpgrade
=== CONT TestKubernetesUpgrade
=== CONT TestKubernetesUpgrade
version_upgrade_test.go:229: (dbg) Run: out/minikube-linux-amd64 start -p kubernetes-upgrade-20220630192547-10455 --memory=2200 --kubernetes-version=v1.16.0 --alsologtostderr -v=1 --driver=docker --container-runtime=containerd
=== CONT TestKubernetesUpgrade
version_upgrade_test.go:229: (dbg) Done: out/minikube-linux-amd64 start -p kubernetes-upgrade-20220630192547-10455 --memory=2200 --kubernetes-version=v1.16.0 --alsologtostderr -v=1 --driver=docker --container-runtime=containerd: (1m0.169937254s)
version_upgrade_test.go:234: (dbg) Run: out/minikube-linux-amd64 stop -p kubernetes-upgrade-20220630192547-10455
version_upgrade_test.go:234: (dbg) Done: out/minikube-linux-amd64 stop -p kubernetes-upgrade-20220630192547-10455: (1.33563467s)
version_upgrade_test.go:239: (dbg) Run: out/minikube-linux-amd64 -p kubernetes-upgrade-20220630192547-10455 status --format={{.Host}}
version_upgrade_test.go:239: (dbg) Non-zero exit: out/minikube-linux-amd64 -p kubernetes-upgrade-20220630192547-10455 status --format={{.Host}}: exit status 7 (113.863651ms)
-- stdout --
Stopped
-- /stdout --
version_upgrade_test.go:241: status error: exit status 7 (may be ok)
version_upgrade_test.go:250: (dbg) Run: out/minikube-linux-amd64 start -p kubernetes-upgrade-20220630192547-10455 --memory=2200 --kubernetes-version=v1.24.2 --alsologtostderr -v=1 --driver=docker --container-runtime=containerd
=== CONT TestKubernetesUpgrade
version_upgrade_test.go:250: (dbg) Non-zero exit: out/minikube-linux-amd64 start -p kubernetes-upgrade-20220630192547-10455 --memory=2200 --kubernetes-version=v1.24.2 --alsologtostderr -v=1 --driver=docker --container-runtime=containerd: exit status 109 (8m31.160314185s)
-- stdout --
* [kubernetes-upgrade-20220630192547-10455] minikube v1.26.0 on Ubuntu 20.04 (kvm/amd64)
- MINIKUBE_LOCATION=14420
- MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
- KUBECONFIG=/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/kubeconfig
- MINIKUBE_HOME=/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube
- MINIKUBE_BIN=out/minikube-linux-amd64
* Using the docker driver based on existing profile
* Starting control plane node kubernetes-upgrade-20220630192547-10455 in cluster kubernetes-upgrade-20220630192547-10455
* Pulling base image ...
* Restarting existing docker container for "kubernetes-upgrade-20220630192547-10455" ...
* Preparing Kubernetes v1.24.2 on containerd 1.6.6 ...
- kubelet.cni-conf-dir=/etc/cni/net.mk
- Generating certificates and keys ...
- Booting up control plane ...
- Generating certificates and keys ...
- Booting up control plane ...
X Problems detected in kubelet:
Jun 30 19:35:20 kubernetes-upgrade-20220630192547-10455 kubelet[11511]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
-- /stdout --
** stderr **
I0630 19:26:49.664517 144400 out.go:296] Setting OutFile to fd 1 ...
I0630 19:26:49.664658 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:26:49.664671 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:26:49.664679 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:26:49.665451 144400 root.go:329] Updating PATH: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/bin
I0630 19:26:49.665959 144400 out.go:303] Setting JSON to false
I0630 19:26:49.668192 144400 start.go:115] hostinfo: {"hostname":"ubuntu-20-agent-10","uptime":4164,"bootTime":1656613046,"procs":728,"os":"linux","platform":"ubuntu","platformFamily":"debian","platformVersion":"20.04","kernelVersion":"5.15.0-1012-gcp","kernelArch":"x86_64","virtualizationSystem":"kvm","virtualizationRole":"guest","hostId":"591c9f12-2938-3743-e2bf-c56a050d43d1"}
I0630 19:26:49.668290 144400 start.go:125] virtualization: kvm guest
I0630 19:26:49.673255 144400 out.go:177] * [kubernetes-upgrade-20220630192547-10455] minikube v1.26.0 on Ubuntu 20.04 (kvm/amd64)
I0630 19:26:49.675740 144400 notify.go:193] Checking for updates...
I0630 19:26:49.675742 144400 out.go:177] - MINIKUBE_LOCATION=14420
I0630 19:26:49.675784 144400 preload.go:306] deleting older generation preload /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v8-v1.20.0-containerd-overlay2-amd64.tar.lz4
I0630 19:26:49.679510 144400 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0630 19:26:49.681051 144400 out.go:177] - KUBECONFIG=/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/kubeconfig
I0630 19:26:49.685594 144400 out.go:177] - MINIKUBE_HOME=/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube
I0630 19:26:49.687374 144400 out.go:177] - MINIKUBE_BIN=out/minikube-linux-amd64
I0630 19:26:49.689484 144400 config.go:178] Loaded profile config "kubernetes-upgrade-20220630192547-10455": Driver=docker, ContainerRuntime=containerd, KubernetesVersion=v1.16.0
I0630 19:26:49.690081 144400 driver.go:360] Setting default libvirt URI to qemu:///system
I0630 19:26:49.736682 144400 docker.go:137] docker version: linux-20.10.17
I0630 19:26:49.736783 144400 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0630 19:26:49.809614 144400 preload.go:306] deleting older generation preload /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v8-v1.20.0-containerd-overlay2-amd64.tar.lz4.checksum
I0630 19:26:49.874213 144400 info.go:265] docker info: {ID:TS6T:UINC:MIYS:RZPA:KS6T:4JQK:7JHN:D6RA:LDP2:MHAE:G32M:C5NQ Containers:4 ContainersRunning:3 ContainersPaused:0 ContainersStopped:1 Images:8 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:79 OomKillDisable:true NGoroutines:79 SystemTime:2022-06-30 19:26:49.783334384 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1012-gcp OperatingSystem:Ubuntu 20.04.4 LTS OSType:linux Architecture:x86_64 IndexServ
erAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:8 MemTotal:33662451712 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ubuntu-20-agent-10 Labels:[] ExperimentalBuild:false ServerVersion:20.10.17 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1 Expected:10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1} RuncCommit:{ID:v1.1.2-0-ga916309 Expected:v1.1.2-0-ga916309} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=default] ProductLicense: Warnings:<nil> ServerErrors:[] Clien
tInfo:{Debug:false Plugins:[map[Experimental:true Name:app Path:/usr/libexec/docker/cli-plugins/docker-app SchemaVersion:0.1.0 ShortDescription:Docker App Vendor:Docker Inc. Version:v0.9.1-beta3] map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.8.2-docker] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.6.0] map[Name:scan Path:/usr/libexec/docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.17.0]] Warnings:<nil>}}
I0630 19:26:49.874308 144400 docker.go:254] overlay module found
I0630 19:26:49.877650 144400 out.go:177] * Using the docker driver based on existing profile
I0630 19:26:49.879083 144400 start.go:284] selected driver: docker
I0630 19:26:49.879098 144400 start.go:808] validating driver "docker" against &{Name:kubernetes-upgrade-20220630192547-10455 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.32-1656350719-14420@sha256:e7b7f38d1a2eba7828afc2c4c3d24e1d391db431976e47aa6dc5c7a6b038ca4e Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:kubernetes-upgrade-20220630192547-
10455 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[{Component:kubelet Key:cni-conf-dir Value:/etc/cni/net.mk}] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.67.2 Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[default-storageclass:true storage-provisioner:true] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262
144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath:}
I0630 19:26:49.879184 144400 start.go:819] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0630 19:26:49.879986 144400 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0630 19:26:49.998436 144400 info.go:265] docker info: {ID:TS6T:UINC:MIYS:RZPA:KS6T:4JQK:7JHN:D6RA:LDP2:MHAE:G32M:C5NQ Containers:4 ContainersRunning:3 ContainersPaused:0 ContainersStopped:1 Images:8 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:79 OomKillDisable:true NGoroutines:79 SystemTime:2022-06-30 19:26:49.908178805 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1012-gcp OperatingSystem:Ubuntu 20.04.4 LTS OSType:linux Architecture:x86_64 IndexServ
erAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:8 MemTotal:33662451712 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ubuntu-20-agent-10 Labels:[] ExperimentalBuild:false ServerVersion:20.10.17 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1 Expected:10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1} RuncCommit:{ID:v1.1.2-0-ga916309 Expected:v1.1.2-0-ga916309} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=default] ProductLicense: Warnings:<nil> ServerErrors:[] Clien
tInfo:{Debug:false Plugins:[map[Experimental:true Name:app Path:/usr/libexec/docker/cli-plugins/docker-app SchemaVersion:0.1.0 ShortDescription:Docker App Vendor:Docker Inc. Version:v0.9.1-beta3] map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.8.2-docker] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.6.0] map[Name:scan Path:/usr/libexec/docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.17.0]] Warnings:<nil>}}
I0630 19:26:49.998769 144400 cni.go:95] Creating CNI manager for ""
I0630 19:26:49.998789 144400 cni.go:162] "docker" driver + containerd runtime found, recommending kindnet
I0630 19:26:49.998803 144400 start_flags.go:310] config:
{Name:kubernetes-upgrade-20220630192547-10455 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.32-1656350719-14420@sha256:e7b7f38d1a2eba7828afc2c4c3d24e1d391db431976e47aa6dc5c7a6b038ca4e Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.24.2 ClusterName:kubernetes-upgrade-20220630192547-10455 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:
[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[{Component:kubelet Key:cni-conf-dir Value:/etc/cni/net.mk}] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.67.2 Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[default-storageclass:true storage-provisioner:true] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: Disab
leOptimizations:false DisableMetrics:false CustomQemuFirmwarePath:}
I0630 19:26:50.002346 144400 out.go:177] * Starting control plane node kubernetes-upgrade-20220630192547-10455 in cluster kubernetes-upgrade-20220630192547-10455
I0630 19:26:50.003817 144400 cache.go:120] Beginning downloading kic base image for docker with containerd
I0630 19:26:50.005248 144400 out.go:177] * Pulling base image ...
I0630 19:26:50.006674 144400 preload.go:132] Checking if preload exists for k8s version v1.24.2 and runtime containerd
I0630 19:26:50.006722 144400 preload.go:148] Found local preload: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.24.2-containerd-overlay2-amd64.tar.lz4
I0630 19:26:50.006737 144400 cache.go:57] Caching tarball of preloaded images
I0630 19:26:50.006775 144400 image.go:75] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.32-1656350719-14420@sha256:e7b7f38d1a2eba7828afc2c4c3d24e1d391db431976e47aa6dc5c7a6b038ca4e in local docker daemon
I0630 19:26:50.007007 144400 preload.go:174] Found /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.24.2-containerd-overlay2-amd64.tar.lz4 in cache, skipping download
I0630 19:26:50.007026 144400 cache.go:60] Finished verifying existence of preloaded tar for v1.24.2 on containerd
I0630 19:26:50.007166 144400 profile.go:148] Saving config to /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/profiles/kubernetes-upgrade-20220630192547-10455/config.json ...
I0630 19:26:50.044282 144400 image.go:79] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.32-1656350719-14420@sha256:e7b7f38d1a2eba7828afc2c4c3d24e1d391db431976e47aa6dc5c7a6b038ca4e in local docker daemon, skipping pull
I0630 19:26:50.044311 144400 cache.go:142] gcr.io/k8s-minikube/kicbase-builds:v0.0.32-1656350719-14420@sha256:e7b7f38d1a2eba7828afc2c4c3d24e1d391db431976e47aa6dc5c7a6b038ca4e exists in daemon, skipping load
I0630 19:26:50.044331 144400 cache.go:208] Successfully downloaded all kic artifacts
I0630 19:26:50.044374 144400 start.go:352] acquiring machines lock for kubernetes-upgrade-20220630192547-10455: {Name:mkc39b36ceebec89a6612ba37d5940abdb41f135 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0630 19:26:50.044487 144400 start.go:356] acquired machines lock for "kubernetes-upgrade-20220630192547-10455" in 74.591µs
I0630 19:26:50.044512 144400 start.go:94] Skipping create...Using existing machine configuration
I0630 19:26:50.044520 144400 fix.go:55] fixHost starting:
I0630 19:26:50.044760 144400 cli_runner.go:164] Run: docker container inspect kubernetes-upgrade-20220630192547-10455 --format={{.State.Status}}
I0630 19:26:50.080369 144400 fix.go:103] recreateIfNeeded on kubernetes-upgrade-20220630192547-10455: state=Stopped err=<nil>
W0630 19:26:50.080424 144400 fix.go:129] unexpected machine state, will restart: <nil>
I0630 19:26:50.083828 144400 out.go:177] * Restarting existing docker container for "kubernetes-upgrade-20220630192547-10455" ...
I0630 19:26:50.085230 144400 cli_runner.go:164] Run: docker start kubernetes-upgrade-20220630192547-10455
I0630 19:26:50.549563 144400 cli_runner.go:164] Run: docker container inspect kubernetes-upgrade-20220630192547-10455 --format={{.State.Status}}
I0630 19:26:50.592759 144400 kic.go:416] container "kubernetes-upgrade-20220630192547-10455" state is running.
I0630 19:26:50.593225 144400 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" kubernetes-upgrade-20220630192547-10455
I0630 19:26:50.628269 144400 profile.go:148] Saving config to /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/profiles/kubernetes-upgrade-20220630192547-10455/config.json ...
I0630 19:26:50.628492 144400 machine.go:88] provisioning docker machine ...
I0630 19:26:50.628516 144400 ubuntu.go:169] provisioning hostname "kubernetes-upgrade-20220630192547-10455"
I0630 19:26:50.628566 144400 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" kubernetes-upgrade-20220630192547-10455
I0630 19:26:50.680456 144400 main.go:134] libmachine: Using SSH client type: native
I0630 19:26:50.680678 144400 main.go:134] libmachine: &{{{<nil> 0 [] [] []} docker [0x7dae00] 0x7dde60 <nil> [] 0s} 127.0.0.1 49309 <nil> <nil>}
I0630 19:26:50.680713 144400 main.go:134] libmachine: About to run SSH command:
sudo hostname kubernetes-upgrade-20220630192547-10455 && echo "kubernetes-upgrade-20220630192547-10455" | sudo tee /etc/hostname
I0630 19:26:50.681314 144400 main.go:134] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:44516->127.0.0.1:49309: read: connection reset by peer
I0630 19:26:53.808317 144400 main.go:134] libmachine: SSH cmd err, output: <nil>: kubernetes-upgrade-20220630192547-10455
I0630 19:26:53.808403 144400 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" kubernetes-upgrade-20220630192547-10455
I0630 19:26:53.855496 144400 main.go:134] libmachine: Using SSH client type: native
I0630 19:26:53.855680 144400 main.go:134] libmachine: &{{{<nil> 0 [] [] []} docker [0x7dae00] 0x7dde60 <nil> [] 0s} 127.0.0.1 49309 <nil> <nil>}
I0630 19:26:53.855722 144400 main.go:134] libmachine: About to run SSH command:
if ! grep -xq '.*\skubernetes-upgrade-20220630192547-10455' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 kubernetes-upgrade-20220630192547-10455/g' /etc/hosts;
else
echo '127.0.1.1 kubernetes-upgrade-20220630192547-10455' | sudo tee -a /etc/hosts;
fi
fi
I0630 19:26:53.986505 144400 main.go:134] libmachine: SSH cmd err, output: <nil>:
I0630 19:26:53.986541 144400 ubuntu.go:175] set auth options {CertDir:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube CaCertPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/ca.pem CaPrivateKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/server.pem ServerKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/server-key.pem ClientKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/c
erts/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube}
I0630 19:26:53.986574 144400 ubuntu.go:177] setting up certificates
I0630 19:26:53.986584 144400 provision.go:83] configureAuth start
I0630 19:26:53.986649 144400 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" kubernetes-upgrade-20220630192547-10455
I0630 19:26:54.022821 144400 provision.go:138] copyHostCerts
I0630 19:26:54.022917 144400 exec_runner.go:144] found /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/ca.pem, removing ...
I0630 19:26:54.022936 144400 exec_runner.go:207] rm: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/ca.pem
I0630 19:26:54.023015 144400 exec_runner.go:151] cp: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/ca.pem --> /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/ca.pem (1082 bytes)
I0630 19:26:54.023146 144400 exec_runner.go:144] found /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cert.pem, removing ...
I0630 19:26:54.023163 144400 exec_runner.go:207] rm: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cert.pem
I0630 19:26:54.023203 144400 exec_runner.go:151] cp: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/cert.pem --> /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cert.pem (1123 bytes)
I0630 19:26:54.023285 144400 exec_runner.go:144] found /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/key.pem, removing ...
I0630 19:26:54.023298 144400 exec_runner.go:207] rm: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/key.pem
I0630 19:26:54.023335 144400 exec_runner.go:151] cp: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/key.pem --> /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/key.pem (1679 bytes)
I0630 19:26:54.023410 144400 provision.go:112] generating server cert: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/server.pem ca-key=/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/ca.pem private-key=/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/ca-key.pem org=jenkins.kubernetes-upgrade-20220630192547-10455 san=[192.168.67.2 127.0.0.1 localhost 127.0.0.1 minikube kubernetes-upgrade-20220630192547-10455]
I0630 19:26:54.298694 144400 provision.go:172] copyRemoteCerts
I0630 19:26:54.298774 144400 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0630 19:26:54.298818 144400 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" kubernetes-upgrade-20220630192547-10455
I0630 19:26:54.336018 144400 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49309 SSHKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/kubernetes-upgrade-20220630192547-10455/id_rsa Username:docker}
I0630 19:26:54.420864 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1082 bytes)
I0630 19:26:54.438731 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/server.pem --> /etc/docker/server.pem (1285 bytes)
I0630 19:26:54.454983 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1675 bytes)
I0630 19:26:54.470924 144400 provision.go:86] duration metric: configureAuth took 484.326038ms
I0630 19:26:54.470947 144400 ubuntu.go:193] setting minikube options for container-runtime
I0630 19:26:54.471131 144400 config.go:178] Loaded profile config "kubernetes-upgrade-20220630192547-10455": Driver=docker, ContainerRuntime=containerd, KubernetesVersion=v1.24.2
I0630 19:26:54.471144 144400 machine.go:91] provisioned docker machine in 3.842636802s
I0630 19:26:54.471152 144400 start.go:306] post-start starting for "kubernetes-upgrade-20220630192547-10455" (driver="docker")
I0630 19:26:54.471166 144400 start.go:316] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0630 19:26:54.471216 144400 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0630 19:26:54.471255 144400 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" kubernetes-upgrade-20220630192547-10455
I0630 19:26:54.502734 144400 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49309 SSHKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/kubernetes-upgrade-20220630192547-10455/id_rsa Username:docker}
I0630 19:26:54.591644 144400 ssh_runner.go:195] Run: cat /etc/os-release
I0630 19:26:54.595870 144400 main.go:134] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0630 19:26:54.595903 144400 main.go:134] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0630 19:26:54.595917 144400 main.go:134] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0630 19:26:54.595925 144400 info.go:137] Remote host: Ubuntu 20.04.4 LTS
I0630 19:26:54.595944 144400 filesync.go:126] Scanning /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/addons for local assets ...
I0630 19:26:54.596019 144400 filesync.go:126] Scanning /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/files for local assets ...
I0630 19:26:54.596122 144400 filesync.go:149] local asset: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/files/etc/ssl/certs/104552.pem -> 104552.pem in /etc/ssl/certs
I0630 19:26:54.596240 144400 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0630 19:26:54.605459 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/files/etc/ssl/certs/104552.pem --> /etc/ssl/certs/104552.pem (1708 bytes)
I0630 19:26:54.625678 144400 start.go:309] post-start completed in 154.506346ms
I0630 19:26:54.625782 144400 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0630 19:26:54.625840 144400 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" kubernetes-upgrade-20220630192547-10455
I0630 19:26:54.667869 144400 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49309 SSHKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/kubernetes-upgrade-20220630192547-10455/id_rsa Username:docker}
I0630 19:26:54.755359 144400 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0630 19:26:54.762794 144400 fix.go:57] fixHost completed within 4.718267864s
I0630 19:26:54.762822 144400 start.go:81] releasing machines lock for "kubernetes-upgrade-20220630192547-10455", held for 4.718318706s
I0630 19:26:54.762915 144400 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" kubernetes-upgrade-20220630192547-10455
I0630 19:26:54.802946 144400 ssh_runner.go:195] Run: systemctl --version
I0630 19:26:54.802999 144400 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" kubernetes-upgrade-20220630192547-10455
I0630 19:26:54.803023 144400 ssh_runner.go:195] Run: curl -sS -m 2 https://k8s.gcr.io/
I0630 19:26:54.803089 144400 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" kubernetes-upgrade-20220630192547-10455
I0630 19:26:54.843823 144400 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49309 SSHKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/kubernetes-upgrade-20220630192547-10455/id_rsa Username:docker}
I0630 19:26:54.845334 144400 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49309 SSHKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/kubernetes-upgrade-20220630192547-10455/id_rsa Username:docker}
I0630 19:26:54.957343 144400 ssh_runner.go:195] Run: sudo systemctl stop -f crio
I0630 19:26:54.971361 144400 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0630 19:26:54.982959 144400 docker.go:179] disabling docker service ...
I0630 19:26:54.983016 144400 ssh_runner.go:195] Run: sudo systemctl stop -f docker.socket
I0630 19:26:54.995727 144400 ssh_runner.go:195] Run: sudo systemctl stop -f docker.service
I0630 19:26:55.006622 144400 ssh_runner.go:195] Run: sudo systemctl disable docker.socket
I0630 19:26:55.094862 144400 ssh_runner.go:195] Run: sudo systemctl mask docker.service
I0630 19:26:55.187410 144400 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service docker
I0630 19:26:55.197319 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0630 19:26:55.209487 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo sed -e 's|^.*sandbox_image = .*$|sandbox_image = "k8s.gcr.io/pause:3.7"|' -i /etc/containerd/config.toml"
I0630 19:26:55.217111 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo sed -e 's|^.*restrict_oom_score_adj = .*$|restrict_oom_score_adj = false|' -i /etc/containerd/config.toml"
I0630 19:26:55.224881 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo sed -e 's|^.*SystemdCgroup = .*$|SystemdCgroup = false|' -i /etc/containerd/config.toml"
I0630 19:26:55.233126 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo sed -e 's|^.*conf_dir = .*$|conf_dir = "/etc/cni/net.mk"|' -i /etc/containerd/config.toml"
I0630 19:26:55.241837 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo sed -e 's|^# imports|imports = ["/etc/containerd/containerd.conf.d/02-containerd.conf"]|' -i /etc/containerd/config.toml"
I0630 19:26:55.249501 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc/containerd/containerd.conf.d && printf %s "dmVyc2lvbiA9IDIK" | base64 -d | sudo tee /etc/containerd/containerd.conf.d/02-containerd.conf"
I0630 19:26:55.262158 144400 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0630 19:26:55.268514 144400 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0630 19:26:55.275680 144400 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0630 19:26:55.362596 144400 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0630 19:26:55.442518 144400 start.go:447] Will wait 60s for socket path /run/containerd/containerd.sock
I0630 19:26:55.442590 144400 ssh_runner.go:195] Run: stat /run/containerd/containerd.sock
I0630 19:26:55.478404 144400 start.go:468] Will wait 60s for crictl version
I0630 19:26:55.478469 144400 ssh_runner.go:195] Run: sudo crictl version
I0630 19:26:55.505441 144400 start.go:477] Version: 0.1.0
RuntimeName: containerd
RuntimeVersion: 1.6.6
RuntimeApiVersion: v1alpha2
I0630 19:26:55.505510 144400 ssh_runner.go:195] Run: containerd --version
I0630 19:26:55.535605 144400 ssh_runner.go:195] Run: containerd --version
I0630 19:26:55.568439 144400 out.go:177] * Preparing Kubernetes v1.24.2 on containerd 1.6.6 ...
I0630 19:26:55.569848 144400 cli_runner.go:164] Run: docker network inspect kubernetes-upgrade-20220630192547-10455 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0630 19:26:55.607131 144400 ssh_runner.go:195] Run: grep 192.168.67.1 host.minikube.internal$ /etc/hosts
I0630 19:26:55.611109 144400 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.67.1 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0630 19:26:55.623478 144400 out.go:177] - kubelet.cni-conf-dir=/etc/cni/net.mk
I0630 19:26:55.625118 144400 preload.go:132] Checking if preload exists for k8s version v1.24.2 and runtime containerd
I0630 19:26:55.625173 144400 ssh_runner.go:195] Run: sudo crictl images --output json
I0630 19:26:55.655568 144400 containerd.go:543] couldn't find preloaded image for "k8s.gcr.io/kube-apiserver:v1.24.2". assuming images are not preloaded.
I0630 19:26:55.655631 144400 ssh_runner.go:195] Run: which lz4
I0630 19:26:55.659840 144400 ssh_runner.go:195] Run: stat -c "%s %y" /preloaded.tar.lz4
I0630 19:26:55.663940 144400 ssh_runner.go:352] existence check for /preloaded.tar.lz4: stat -c "%s %y" /preloaded.tar.lz4: Process exited with status 1
stdout:
stderr:
stat: cannot stat '/preloaded.tar.lz4': No such file or directory
I0630 19:26:55.663978 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.24.2-containerd-overlay2-amd64.tar.lz4 --> /preloaded.tar.lz4 (447741112 bytes)
I0630 19:26:56.989570 144400 containerd.go:490] Took 1.329769 seconds to copy over tarball
I0630 19:26:56.989646 144400 ssh_runner.go:195] Run: sudo tar -I lz4 -C /var -xf /preloaded.tar.lz4
I0630 19:26:59.526668 144400 ssh_runner.go:235] Completed: sudo tar -I lz4 -C /var -xf /preloaded.tar.lz4: (2.536993031s)
I0630 19:26:59.526692 144400 containerd.go:497] Took 2.537091 seconds t extract the tarball
I0630 19:26:59.526702 144400 ssh_runner.go:146] rm: /preloaded.tar.lz4
I0630 19:26:59.592073 144400 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0630 19:26:59.704559 144400 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0630 19:26:59.976545 144400 ssh_runner.go:195] Run: sudo crictl images --output json
I0630 19:27:00.002027 144400 cache_images.go:88] LoadImages start: [k8s.gcr.io/kube-apiserver:v1.24.2 k8s.gcr.io/kube-controller-manager:v1.24.2 k8s.gcr.io/kube-scheduler:v1.24.2 k8s.gcr.io/kube-proxy:v1.24.2 k8s.gcr.io/pause:3.7 k8s.gcr.io/etcd:3.5.3-0 k8s.gcr.io/coredns/coredns:v1.8.6 gcr.io/k8s-minikube/storage-provisioner:v5]
I0630 19:27:00.002113 144400 image.go:134] retrieving image: gcr.io/k8s-minikube/storage-provisioner:v5
I0630 19:27:00.002128 144400 image.go:134] retrieving image: k8s.gcr.io/kube-proxy:v1.24.2
I0630 19:27:00.002166 144400 image.go:134] retrieving image: k8s.gcr.io/kube-controller-manager:v1.24.2
I0630 19:27:00.002165 144400 image.go:134] retrieving image: k8s.gcr.io/pause:3.7
I0630 19:27:00.002188 144400 image.go:134] retrieving image: k8s.gcr.io/kube-apiserver:v1.24.2
I0630 19:27:00.002213 144400 image.go:134] retrieving image: k8s.gcr.io/kube-scheduler:v1.24.2
I0630 19:27:00.002233 144400 image.go:134] retrieving image: k8s.gcr.io/coredns/coredns:v1.8.6
I0630 19:27:00.002138 144400 image.go:134] retrieving image: k8s.gcr.io/etcd:3.5.3-0
I0630 19:27:00.003293 144400 image.go:177] daemon lookup for k8s.gcr.io/kube-apiserver:v1.24.2: Error: No such image: k8s.gcr.io/kube-apiserver:v1.24.2
I0630 19:27:00.003305 144400 image.go:177] daemon lookup for k8s.gcr.io/kube-controller-manager:v1.24.2: Error: No such image: k8s.gcr.io/kube-controller-manager:v1.24.2
I0630 19:27:00.003346 144400 image.go:177] daemon lookup for k8s.gcr.io/kube-proxy:v1.24.2: Error: No such image: k8s.gcr.io/kube-proxy:v1.24.2
I0630 19:27:00.003367 144400 image.go:177] daemon lookup for k8s.gcr.io/coredns/coredns:v1.8.6: Error: No such image: k8s.gcr.io/coredns/coredns:v1.8.6
I0630 19:27:00.003294 144400 image.go:177] daemon lookup for k8s.gcr.io/etcd:3.5.3-0: Error: No such image: k8s.gcr.io/etcd:3.5.3-0
I0630 19:27:00.003351 144400 image.go:177] daemon lookup for k8s.gcr.io/pause:3.7: Error: No such image: k8s.gcr.io/pause:3.7
I0630 19:27:00.003294 144400 image.go:177] daemon lookup for k8s.gcr.io/kube-scheduler:v1.24.2: Error: No such image: k8s.gcr.io/kube-scheduler:v1.24.2
I0630 19:27:00.003298 144400 image.go:177] daemon lookup for gcr.io/k8s-minikube/storage-provisioner:v5: Error: No such image: gcr.io/k8s-minikube/storage-provisioner:v5
I0630 19:27:00.401873 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo ctr -n=k8s.io images check | grep k8s.gcr.io/pause:3.7"
I0630 19:27:00.401904 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo ctr -n=k8s.io images check | grep k8s.gcr.io/kube-scheduler:v1.24.2"
I0630 19:27:00.406641 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo ctr -n=k8s.io images check | grep k8s.gcr.io/kube-apiserver:v1.24.2"
I0630 19:27:00.409026 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo ctr -n=k8s.io images check | grep k8s.gcr.io/kube-proxy:v1.24.2"
I0630 19:27:00.449244 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo ctr -n=k8s.io images check | grep k8s.gcr.io/kube-controller-manager:v1.24.2"
I0630 19:27:00.465950 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo ctr -n=k8s.io images check | grep k8s.gcr.io/coredns/coredns:v1.8.6"
I0630 19:27:00.536710 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo ctr -n=k8s.io images check | grep k8s.gcr.io/etcd:3.5.3-0"
I0630 19:27:00.777101 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo ctr -n=k8s.io images check | grep gcr.io/k8s-minikube/storage-provisioner:v5"
I0630 19:27:01.147553 144400 cache_images.go:116] "k8s.gcr.io/pause:3.7" needs transfer: "k8s.gcr.io/pause:3.7" does not exist at hash "221177c6082a88ea4f6240ab2450d540955ac6f4d5454f0e15751b653ebda165" in container runtime
I0630 19:27:01.147584 144400 cache_images.go:116] "k8s.gcr.io/kube-scheduler:v1.24.2" needs transfer: "k8s.gcr.io/kube-scheduler:v1.24.2" does not exist at hash "5d725196c1f47e72d2bc7069776d5928b1fb1e4adf09c18997733099aa3663ac" in container runtime
I0630 19:27:01.147606 144400 cri.go:216] Removing image: k8s.gcr.io/pause:3.7
I0630 19:27:01.147612 144400 cri.go:216] Removing image: k8s.gcr.io/kube-scheduler:v1.24.2
I0630 19:27:01.147640 144400 ssh_runner.go:195] Run: which crictl
I0630 19:27:01.147643 144400 ssh_runner.go:195] Run: which crictl
I0630 19:27:01.157709 144400 cache_images.go:116] "k8s.gcr.io/kube-apiserver:v1.24.2" needs transfer: "k8s.gcr.io/kube-apiserver:v1.24.2" does not exist at hash "d3377ffb7177cc4becce8a534d8547aca9530cb30fac9ebe479b31102f1ba503" in container runtime
I0630 19:27:01.157771 144400 cri.go:216] Removing image: k8s.gcr.io/kube-apiserver:v1.24.2
I0630 19:27:01.157810 144400 ssh_runner.go:195] Run: which crictl
I0630 19:27:01.157925 144400 cache_images.go:116] "k8s.gcr.io/kube-proxy:v1.24.2" needs transfer: "k8s.gcr.io/kube-proxy:v1.24.2" does not exist at hash "a634548d10b032c2a1d704ef9a2ab04c12b0574afe67ee192b196a7f12da9536" in container runtime
I0630 19:27:01.157956 144400 cri.go:216] Removing image: k8s.gcr.io/kube-proxy:v1.24.2
I0630 19:27:01.157985 144400 ssh_runner.go:195] Run: which crictl
I0630 19:27:01.330082 144400 cache_images.go:116] "k8s.gcr.io/kube-controller-manager:v1.24.2" needs transfer: "k8s.gcr.io/kube-controller-manager:v1.24.2" does not exist at hash "34cdf99b1bb3b3a62c5b4226c3bc0983ab1f13e776269d1872092091b07203df" in container runtime
I0630 19:27:01.330133 144400 cri.go:216] Removing image: k8s.gcr.io/kube-controller-manager:v1.24.2
I0630 19:27:01.330185 144400 ssh_runner.go:195] Run: which crictl
I0630 19:27:01.333262 144400 cache_images.go:116] "k8s.gcr.io/coredns/coredns:v1.8.6" needs transfer: "k8s.gcr.io/coredns/coredns:v1.8.6" does not exist at hash "a4ca41631cc7ac19ce1be3ebf0314ac5f47af7c711f17066006db82ee3b75b03" in container runtime
I0630 19:27:01.333300 144400 cri.go:216] Removing image: k8s.gcr.io/coredns/coredns:v1.8.6
I0630 19:27:01.333334 144400 ssh_runner.go:195] Run: which crictl
I0630 19:27:01.357360 144400 cache_images.go:116] "k8s.gcr.io/etcd:3.5.3-0" needs transfer: "k8s.gcr.io/etcd:3.5.3-0" does not exist at hash "aebe758cef4cd05b9f8cee39758227714d02f42ef3088023c1e3cd454f927a2b" in container runtime
I0630 19:27:01.357410 144400 cri.go:216] Removing image: k8s.gcr.io/etcd:3.5.3-0
I0630 19:27:01.357452 144400 ssh_runner.go:195] Run: which crictl
I0630 19:27:01.442365 144400 cache_images.go:116] "gcr.io/k8s-minikube/storage-provisioner:v5" needs transfer: "gcr.io/k8s-minikube/storage-provisioner:v5" does not exist at hash "6e38f40d628db3002f5617342c8872c935de530d867d0f709a2fbda1a302a562" in container runtime
I0630 19:27:01.442414 144400 cri.go:216] Removing image: gcr.io/k8s-minikube/storage-provisioner:v5
I0630 19:27:01.442444 144400 ssh_runner.go:195] Run: which crictl
I0630 19:27:01.442479 144400 ssh_runner.go:195] Run: sudo /usr/bin/crictl rmi k8s.gcr.io/pause:3.7
I0630 19:27:01.442508 144400 ssh_runner.go:195] Run: sudo /usr/bin/crictl rmi k8s.gcr.io/kube-proxy:v1.24.2
I0630 19:27:01.442481 144400 ssh_runner.go:195] Run: sudo /usr/bin/crictl rmi k8s.gcr.io/kube-apiserver:v1.24.2
I0630 19:27:01.442555 144400 ssh_runner.go:195] Run: sudo /usr/bin/crictl rmi k8s.gcr.io/kube-controller-manager:v1.24.2
I0630 19:27:01.442444 144400 ssh_runner.go:195] Run: sudo /usr/bin/crictl rmi k8s.gcr.io/kube-scheduler:v1.24.2
I0630 19:27:01.442600 144400 ssh_runner.go:195] Run: sudo /usr/bin/crictl rmi k8s.gcr.io/coredns/coredns:v1.8.6
I0630 19:27:01.442626 144400 ssh_runner.go:195] Run: sudo /usr/bin/crictl rmi k8s.gcr.io/etcd:3.5.3-0
I0630 19:27:03.344678 144400 ssh_runner.go:235] Completed: sudo /usr/bin/crictl rmi k8s.gcr.io/pause:3.7: (1.902166222s)
I0630 19:27:03.344709 144400 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/pause_3.7
I0630 19:27:03.344738 144400 ssh_runner.go:235] Completed: sudo /usr/bin/crictl rmi k8s.gcr.io/kube-apiserver:v1.24.2: (1.902183905s)
I0630 19:27:03.344755 144400 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/kube-apiserver_v1.24.2
I0630 19:27:03.344794 144400 ssh_runner.go:195] Run: stat -c "%s %y" /var/lib/minikube/images/pause_3.7
I0630 19:27:03.344829 144400 ssh_runner.go:195] Run: stat -c "%s %y" /var/lib/minikube/images/kube-apiserver_v1.24.2
I0630 19:27:03.344856 144400 ssh_runner.go:235] Completed: sudo /usr/bin/crictl rmi k8s.gcr.io/kube-proxy:v1.24.2: (1.902330882s)
I0630 19:27:03.344863 144400 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/kube-proxy_v1.24.2
I0630 19:27:03.344904 144400 ssh_runner.go:195] Run: stat -c "%s %y" /var/lib/minikube/images/kube-proxy_v1.24.2
I0630 19:27:03.344914 144400 ssh_runner.go:235] Completed: which crictl: (1.902455275s)
I0630 19:27:03.344943 144400 ssh_runner.go:195] Run: sudo /usr/bin/crictl rmi gcr.io/k8s-minikube/storage-provisioner:v5
I0630 19:27:03.344962 144400 ssh_runner.go:235] Completed: sudo /usr/bin/crictl rmi k8s.gcr.io/kube-controller-manager:v1.24.2: (1.90239176s)
I0630 19:27:03.344968 144400 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/kube-controller-manager_v1.24.2
I0630 19:27:03.345009 144400 ssh_runner.go:235] Completed: sudo /usr/bin/crictl rmi k8s.gcr.io/kube-scheduler:v1.24.2: (1.902426008s)
I0630 19:27:03.345012 144400 ssh_runner.go:195] Run: stat -c "%s %y" /var/lib/minikube/images/kube-controller-manager_v1.24.2
I0630 19:27:03.345017 144400 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/kube-scheduler_v1.24.2
I0630 19:27:03.345064 144400 ssh_runner.go:195] Run: stat -c "%s %y" /var/lib/minikube/images/kube-scheduler_v1.24.2
I0630 19:27:03.345067 144400 ssh_runner.go:235] Completed: sudo /usr/bin/crictl rmi k8s.gcr.io/etcd:3.5.3-0: (1.902429083s)
I0630 19:27:03.345074 144400 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/etcd_3.5.3-0
I0630 19:27:03.345112 144400 ssh_runner.go:195] Run: stat -c "%s %y" /var/lib/minikube/images/etcd_3.5.3-0
I0630 19:27:03.345115 144400 ssh_runner.go:235] Completed: sudo /usr/bin/crictl rmi k8s.gcr.io/coredns/coredns:v1.8.6: (1.902495591s)
I0630 19:27:03.345123 144400 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/coredns/coredns_v1.8.6
I0630 19:27:03.345168 144400 ssh_runner.go:195] Run: stat -c "%s %y" /var/lib/minikube/images/coredns_v1.8.6
I0630 19:27:03.353020 144400 ssh_runner.go:352] existence check for /var/lib/minikube/images/coredns_v1.8.6: stat -c "%s %y" /var/lib/minikube/images/coredns_v1.8.6: Process exited with status 1
stdout:
stderr:
stat: cannot stat '/var/lib/minikube/images/coredns_v1.8.6': No such file or directory
I0630 19:27:03.353051 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/coredns/coredns_v1.8.6 --> /var/lib/minikube/images/coredns_v1.8.6 (13586432 bytes)
I0630 19:27:03.396730 144400 containerd.go:227] Loading image: /var/lib/minikube/images/coredns_v1.8.6
I0630 19:27:03.396803 144400 ssh_runner.go:195] Run: sudo ctr -n=k8s.io images import /var/lib/minikube/images/coredns_v1.8.6
I0630 19:27:03.544256 144400 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/gcr.io/k8s-minikube/storage-provisioner_v5
I0630 19:27:03.544363 144400 ssh_runner.go:195] Run: stat -c "%s %y" /var/lib/minikube/images/storage-provisioner_v5
I0630 19:27:03.544384 144400 ssh_runner.go:352] existence check for /var/lib/minikube/images/kube-apiserver_v1.24.2: stat -c "%s %y" /var/lib/minikube/images/kube-apiserver_v1.24.2: Process exited with status 1
stdout:
stderr:
stat: cannot stat '/var/lib/minikube/images/kube-apiserver_v1.24.2': No such file or directory
I0630 19:27:03.544417 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/kube-apiserver_v1.24.2 --> /var/lib/minikube/images/kube-apiserver_v1.24.2 (33798144 bytes)
I0630 19:27:03.544431 144400 ssh_runner.go:352] existence check for /var/lib/minikube/images/pause_3.7: stat -c "%s %y" /var/lib/minikube/images/pause_3.7: Process exited with status 1
stdout:
stderr:
stat: cannot stat '/var/lib/minikube/images/pause_3.7': No such file or directory
I0630 19:27:03.544464 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/pause_3.7 --> /var/lib/minikube/images/pause_3.7 (311296 bytes)
I0630 19:27:03.544479 144400 ssh_runner.go:352] existence check for /var/lib/minikube/images/etcd_3.5.3-0: stat -c "%s %y" /var/lib/minikube/images/etcd_3.5.3-0: Process exited with status 1
stdout:
stderr:
stat: cannot stat '/var/lib/minikube/images/etcd_3.5.3-0': No such file or directory
I0630 19:27:03.544507 144400 ssh_runner.go:352] existence check for /var/lib/minikube/images/kube-scheduler_v1.24.2: stat -c "%s %y" /var/lib/minikube/images/kube-scheduler_v1.24.2: Process exited with status 1
stdout:
stderr:
stat: cannot stat '/var/lib/minikube/images/kube-scheduler_v1.24.2': No such file or directory
I0630 19:27:03.544508 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/etcd_3.5.3-0 --> /var/lib/minikube/images/etcd_3.5.3-0 (102146048 bytes)
I0630 19:27:03.544527 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/kube-scheduler_v1.24.2 --> /var/lib/minikube/images/kube-scheduler_v1.24.2 (15491584 bytes)
I0630 19:27:03.544569 144400 ssh_runner.go:352] existence check for /var/lib/minikube/images/kube-controller-manager_v1.24.2: stat -c "%s %y" /var/lib/minikube/images/kube-controller-manager_v1.24.2: Process exited with status 1
stdout:
stderr:
stat: cannot stat '/var/lib/minikube/images/kube-controller-manager_v1.24.2': No such file or directory
I0630 19:27:03.544594 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/kube-controller-manager_v1.24.2 --> /var/lib/minikube/images/kube-controller-manager_v1.24.2 (31037952 bytes)
I0630 19:27:03.544658 144400 ssh_runner.go:352] existence check for /var/lib/minikube/images/kube-proxy_v1.24.2: stat -c "%s %y" /var/lib/minikube/images/kube-proxy_v1.24.2: Process exited with status 1
stdout:
stderr:
stat: cannot stat '/var/lib/minikube/images/kube-proxy_v1.24.2': No such file or directory
I0630 19:27:03.544689 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/kube-proxy_v1.24.2 --> /var/lib/minikube/images/kube-proxy_v1.24.2 (39518208 bytes)
I0630 19:27:05.503783 144400 ssh_runner.go:235] Completed: sudo ctr -n=k8s.io images import /var/lib/minikube/images/coredns_v1.8.6: (2.106954283s)
I0630 19:27:05.503815 144400 cache_images.go:315] Transferred and loaded /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/coredns/coredns_v1.8.6 from cache
I0630 19:27:05.503843 144400 containerd.go:227] Loading image: /var/lib/minikube/images/pause_3.7
I0630 19:27:05.503864 144400 ssh_runner.go:235] Completed: stat -c "%s %y" /var/lib/minikube/images/storage-provisioner_v5: (1.959473511s)
I0630 19:27:05.503887 144400 ssh_runner.go:195] Run: sudo ctr -n=k8s.io images import /var/lib/minikube/images/pause_3.7
I0630 19:27:05.503894 144400 ssh_runner.go:352] existence check for /var/lib/minikube/images/storage-provisioner_v5: stat -c "%s %y" /var/lib/minikube/images/storage-provisioner_v5: Process exited with status 1
stdout:
stderr:
stat: cannot stat '/var/lib/minikube/images/storage-provisioner_v5': No such file or directory
I0630 19:27:05.503930 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/gcr.io/k8s-minikube/storage-provisioner_v5 --> /var/lib/minikube/images/storage-provisioner_v5 (9060352 bytes)
I0630 19:27:05.632070 144400 cache_images.go:315] Transferred and loaded /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/pause_3.7 from cache
I0630 19:27:05.632106 144400 containerd.go:227] Loading image: /var/lib/minikube/images/kube-scheduler_v1.24.2
I0630 19:27:05.632143 144400 ssh_runner.go:195] Run: sudo ctr -n=k8s.io images import /var/lib/minikube/images/kube-scheduler_v1.24.2
I0630 19:27:06.477280 144400 cache_images.go:315] Transferred and loaded /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/kube-scheduler_v1.24.2 from cache
I0630 19:27:06.477330 144400 containerd.go:227] Loading image: /var/lib/minikube/images/kube-controller-manager_v1.24.2
I0630 19:27:06.477401 144400 ssh_runner.go:195] Run: sudo ctr -n=k8s.io images import /var/lib/minikube/images/kube-controller-manager_v1.24.2
I0630 19:27:07.851393 144400 ssh_runner.go:235] Completed: sudo ctr -n=k8s.io images import /var/lib/minikube/images/kube-controller-manager_v1.24.2: (1.373962501s)
I0630 19:27:07.851422 144400 cache_images.go:315] Transferred and loaded /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/kube-controller-manager_v1.24.2 from cache
I0630 19:27:07.851449 144400 containerd.go:227] Loading image: /var/lib/minikube/images/kube-apiserver_v1.24.2
I0630 19:27:07.851492 144400 ssh_runner.go:195] Run: sudo ctr -n=k8s.io images import /var/lib/minikube/images/kube-apiserver_v1.24.2
I0630 19:27:12.251316 144400 ssh_runner.go:235] Completed: sudo ctr -n=k8s.io images import /var/lib/minikube/images/kube-apiserver_v1.24.2: (4.399776835s)
I0630 19:27:12.251351 144400 cache_images.go:315] Transferred and loaded /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/kube-apiserver_v1.24.2 from cache
I0630 19:27:12.251396 144400 containerd.go:227] Loading image: /var/lib/minikube/images/kube-proxy_v1.24.2
I0630 19:27:12.251470 144400 ssh_runner.go:195] Run: sudo ctr -n=k8s.io images import /var/lib/minikube/images/kube-proxy_v1.24.2
I0630 19:27:13.191154 144400 cache_images.go:315] Transferred and loaded /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/kube-proxy_v1.24.2 from cache
I0630 19:27:13.191207 144400 containerd.go:227] Loading image: /var/lib/minikube/images/etcd_3.5.3-0
I0630 19:27:13.191267 144400 ssh_runner.go:195] Run: sudo ctr -n=k8s.io images import /var/lib/minikube/images/etcd_3.5.3-0
I0630 19:27:19.609156 144400 ssh_runner.go:235] Completed: sudo ctr -n=k8s.io images import /var/lib/minikube/images/etcd_3.5.3-0: (6.417849497s)
I0630 19:27:19.609193 144400 cache_images.go:315] Transferred and loaded /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/k8s.gcr.io/etcd_3.5.3-0 from cache
I0630 19:27:19.609215 144400 containerd.go:227] Loading image: /var/lib/minikube/images/storage-provisioner_v5
I0630 19:27:19.609262 144400 ssh_runner.go:195] Run: sudo ctr -n=k8s.io images import /var/lib/minikube/images/storage-provisioner_v5
I0630 19:27:22.215487 144400 ssh_runner.go:235] Completed: sudo ctr -n=k8s.io images import /var/lib/minikube/images/storage-provisioner_v5: (2.606200224s)
I0630 19:27:22.215510 144400 cache_images.go:315] Transferred and loaded /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/images/amd64/gcr.io/k8s-minikube/storage-provisioner_v5 from cache
I0630 19:27:22.215531 144400 cache_images.go:123] Successfully loaded all cached images
I0630 19:27:22.215537 144400 cache_images.go:92] LoadImages completed in 22.213485326s
I0630 19:27:22.215587 144400 ssh_runner.go:195] Run: sudo crictl info
I0630 19:27:22.246994 144400 cni.go:95] Creating CNI manager for ""
I0630 19:27:22.247018 144400 cni.go:162] "docker" driver + containerd runtime found, recommending kindnet
I0630 19:27:22.247036 144400 kubeadm.go:87] Using pod CIDR: 10.244.0.0/16
I0630 19:27:22.247053 144400 kubeadm.go:158] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.67.2 APIServerPort:8443 KubernetesVersion:v1.24.2 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:kubernetes-upgrade-20220630192547-10455 NodeName:kubernetes-upgrade-20220630192547-10455 DNSDomain:cluster.local CRISocket:/run/containerd/containerd.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.67.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NoTaintMaster:true NodeIP:192.168.67.2 CgroupDriver:
cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticPodPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[]}
I0630 19:27:22.247197 144400 kubeadm.go:162] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.67.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: /run/containerd/containerd.sock
name: "kubernetes-upgrade-20220630192547-10455"
kubeletExtraArgs:
node-ip: 192.168.67.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.67.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: mk
controlPlaneEndpoint: control-plane.minikube.internal:8443
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
proxy-refresh-interval: "70000"
kubernetesVersion: v1.24.2
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0630 19:27:22.247285 144400 kubeadm.go:961] kubelet [Unit]
Wants=containerd.service
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.24.2/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --cni-conf-dir=/etc/cni/net.mk --config=/var/lib/kubelet/config.yaml --container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock --hostname-override=kubernetes-upgrade-20220630192547-10455 --image-service-endpoint=unix:///run/containerd/containerd.sock --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.67.2 --runtime-request-timeout=15m
[Install]
config:
{KubernetesVersion:v1.24.2 ClusterName:kubernetes-upgrade-20220630192547-10455 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[{Component:kubelet Key:cni-conf-dir Value:/etc/cni/net.mk}] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:}
I0630 19:27:22.247338 144400 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.24.2
I0630 19:27:22.254828 144400 binaries.go:44] Found k8s binaries, skipping transfer
I0630 19:27:22.254892 144400 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0630 19:27:22.262543 144400 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (563 bytes)
I0630 19:27:22.274576 144400 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0630 19:27:22.286108 144400 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2068 bytes)
I0630 19:27:22.298204 144400 ssh_runner.go:195] Run: grep 192.168.67.2 control-plane.minikube.internal$ /etc/hosts
I0630 19:27:22.301526 144400 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.67.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0630 19:27:22.312363 144400 certs.go:54] Setting up /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/profiles/kubernetes-upgrade-20220630192547-10455 for IP: 192.168.67.2
I0630 19:27:22.312466 144400 certs.go:182] skipping minikubeCA CA generation: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/ca.key
I0630 19:27:22.312517 144400 certs.go:182] skipping proxyClientCA CA generation: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/proxy-client-ca.key
I0630 19:27:22.312609 144400 certs.go:298] skipping minikube-user signed cert generation: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/profiles/kubernetes-upgrade-20220630192547-10455/client.key
I0630 19:27:22.312672 144400 certs.go:298] skipping minikube signed cert generation: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/profiles/kubernetes-upgrade-20220630192547-10455/apiserver.key.c7fa3a9e
I0630 19:27:22.312711 144400 certs.go:298] skipping aggregator signed cert generation: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/profiles/kubernetes-upgrade-20220630192547-10455/proxy-client.key
I0630 19:27:22.312830 144400 certs.go:388] found cert: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/10455.pem (1338 bytes)
W0630 19:27:22.312875 144400 certs.go:384] ignoring /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/10455_empty.pem, impossibly tiny 0 bytes
I0630 19:27:22.312893 144400 certs.go:388] found cert: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/ca-key.pem (1679 bytes)
I0630 19:27:22.312930 144400 certs.go:388] found cert: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/ca.pem (1082 bytes)
I0630 19:27:22.312971 144400 certs.go:388] found cert: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/cert.pem (1123 bytes)
I0630 19:27:22.313007 144400 certs.go:388] found cert: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/key.pem (1679 bytes)
I0630 19:27:22.313061 144400 certs.go:388] found cert: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/files/etc/ssl/certs/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/files/etc/ssl/certs/104552.pem (1708 bytes)
I0630 19:27:22.313666 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/profiles/kubernetes-upgrade-20220630192547-10455/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1399 bytes)
I0630 19:27:22.334068 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/profiles/kubernetes-upgrade-20220630192547-10455/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1679 bytes)
I0630 19:27:22.351347 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/profiles/kubernetes-upgrade-20220630192547-10455/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0630 19:27:22.367230 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/profiles/kubernetes-upgrade-20220630192547-10455/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1675 bytes)
I0630 19:27:22.383090 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0630 19:27:22.399166 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1675 bytes)
I0630 19:27:22.415425 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0630 19:27:22.431722 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1679 bytes)
I0630 19:27:22.448573 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0630 19:27:22.467154 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/10455.pem --> /usr/share/ca-certificates/10455.pem (1338 bytes)
I0630 19:27:22.486065 144400 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/files/etc/ssl/certs/104552.pem --> /usr/share/ca-certificates/104552.pem (1708 bytes)
I0630 19:27:22.502793 144400 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0630 19:27:22.514831 144400 ssh_runner.go:195] Run: openssl version
I0630 19:27:22.519623 144400 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0630 19:27:22.526548 144400 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0630 19:27:22.529604 144400 certs.go:431] hashing: -rw-r--r-- 1 root root 1111 Jun 30 18:59 /usr/share/ca-certificates/minikubeCA.pem
I0630 19:27:22.529651 144400 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0630 19:27:22.534502 144400 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0630 19:27:22.540929 144400 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/10455.pem && ln -fs /usr/share/ca-certificates/10455.pem /etc/ssl/certs/10455.pem"
I0630 19:27:22.548568 144400 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/10455.pem
I0630 19:27:22.552342 144400 certs.go:431] hashing: -rw-r--r-- 1 root root 1338 Jun 30 19:02 /usr/share/ca-certificates/10455.pem
I0630 19:27:22.552387 144400 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/10455.pem
I0630 19:27:22.557495 144400 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/10455.pem /etc/ssl/certs/51391683.0"
I0630 19:27:22.565329 144400 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/104552.pem && ln -fs /usr/share/ca-certificates/104552.pem /etc/ssl/certs/104552.pem"
I0630 19:27:22.572811 144400 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/104552.pem
I0630 19:27:22.576022 144400 certs.go:431] hashing: -rw-r--r-- 1 root root 1708 Jun 30 19:02 /usr/share/ca-certificates/104552.pem
I0630 19:27:22.576061 144400 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/104552.pem
I0630 19:27:22.580691 144400 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/104552.pem /etc/ssl/certs/3ec20f2e.0"
I0630 19:27:22.587409 144400 kubeadm.go:395] StartCluster: {Name:kubernetes-upgrade-20220630192547-10455 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.32-1656350719-14420@sha256:e7b7f38d1a2eba7828afc2c4c3d24e1d391db431976e47aa6dc5c7a6b038ca4e Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.24.2 ClusterName:kubernetes-upgrade-20220630192547-10455 Namespace:defa
ult APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[{Component:kubelet Key:cni-conf-dir Value:/etc/cni/net.mk}] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.67.2 Port:8443 KubernetesVersion:v1.24.2 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[default-storageclass:true storage-provisioner:true] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[]
MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath:}
I0630 19:27:22.587511 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:paused Name: Namespaces:[kube-system]}
I0630 19:27:22.587567 144400 ssh_runner.go:195] Run: sudo -s eval "crictl ps -a --quiet --label io.kubernetes.pod.namespace=kube-system"
I0630 19:27:22.610329 144400 cri.go:87] found id: ""
I0630 19:27:22.610383 144400 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0630 19:27:22.617477 144400 kubeadm.go:410] found existing configuration files, will attempt cluster restart
I0630 19:27:22.617513 144400 kubeadm.go:626] restartCluster start
I0630 19:27:22.617554 144400 ssh_runner.go:195] Run: sudo test -d /data/minikube
I0630 19:27:22.624135 144400 kubeadm.go:127] /data/minikube skipping compat symlinks: sudo test -d /data/minikube: Process exited with status 1
stdout:
stderr:
I0630 19:27:22.624528 144400 kubeconfig.go:116] verify returned: extract IP: "kubernetes-upgrade-20220630192547-10455" does not appear in /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/kubeconfig
I0630 19:27:22.624624 144400 kubeconfig.go:127] "kubernetes-upgrade-20220630192547-10455" context is missing from /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/kubeconfig - will repair!
I0630 19:27:22.624884 144400 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/kubeconfig: {Name:mkb1b2c7676bb58a79042c55304ccfa61fbe00de Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0630 19:27:22.625417 144400 kapi.go:59] client config for kubernetes-upgrade-20220630192547-10455: &rest.Config{Host:"https://192.168.67.2:8443", APIPath:"", ContentConfig:rest.ContentConfig{AcceptContentTypes:"", ContentType:"", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:runtime.NegotiatedSerializer(nil)}, Username:"", Password:"", BearerToken:"", BearerTokenFile:"", Impersonate:rest.ImpersonationConfig{UserName:"", UID:"", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:<nil>, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:<nil>, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:"", CertFile:"/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/profiles/kubernetes-upgrade-20220630192547-10455/client.crt", KeyFile:"/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikub
e/profiles/kubernetes-upgrade-20220630192547-10455/client.key", CAFile:"/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/ca.crt", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8(nil), NextProtos:[]string(nil)}, UserAgent:"", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(0x173d420), QPS:0, Burst:0, RateLimiter:flowcontrol.RateLimiter(nil), WarningHandler:rest.WarningHandler(nil), Timeout:0, Dial:(func(context.Context, string, string) (net.Conn, error))(nil), Proxy:(func(*http.Request) (*url.URL, error))(nil)}
I0630 19:27:22.625914 144400 ssh_runner.go:195] Run: sudo diff -u /var/tmp/minikube/kubeadm.yaml /var/tmp/minikube/kubeadm.yaml.new
I0630 19:27:22.633991 144400 kubeadm.go:593] needs reconfigure: configs differ:
-- stdout --
--- /var/tmp/minikube/kubeadm.yaml 2022-06-30 19:26:14.162198462 +0000
+++ /var/tmp/minikube/kubeadm.yaml.new 2022-06-30 19:27:22.291378633 +0000
@@ -1,4 +1,4 @@
-apiVersion: kubeadm.k8s.io/v1beta1
+apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.67.2
@@ -17,7 +17,7 @@
node-ip: 192.168.67.2
taints: []
---
-apiVersion: kubeadm.k8s.io/v1beta1
+apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.67.2"]
@@ -31,16 +31,14 @@
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
-clusterName: kubernetes-upgrade-20220630192547-10455
+clusterName: mk
controlPlaneEndpoint: control-plane.minikube.internal:8443
-dns:
- type: CoreDNS
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
- listen-metrics-urls: http://127.0.0.1:2381,http://192.168.67.2:2381
-kubernetesVersion: v1.16.0
+ proxy-refresh-interval: "70000"
+kubernetesVersion: v1.24.2
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
-- /stdout --
I0630 19:27:22.634012 144400 kubeadm.go:1092] stopping kube-system containers ...
I0630 19:27:22.634025 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name: Namespaces:[kube-system]}
I0630 19:27:22.634066 144400 ssh_runner.go:195] Run: sudo -s eval "crictl ps -a --quiet --label io.kubernetes.pod.namespace=kube-system"
I0630 19:27:22.660293 144400 cri.go:87] found id: ""
I0630 19:27:22.660364 144400 ssh_runner.go:195] Run: sudo systemctl stop kubelet
I0630 19:27:22.674619 144400 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0630 19:27:22.682582 144400 kubeadm.go:155] found existing configuration files:
-rw------- 1 root root 5763 Jun 30 19:26 /etc/kubernetes/admin.conf
-rw------- 1 root root 5799 Jun 30 19:26 /etc/kubernetes/controller-manager.conf
-rw------- 1 root root 5963 Jun 30 19:26 /etc/kubernetes/kubelet.conf
-rw------- 1 root root 5743 Jun 30 19:26 /etc/kubernetes/scheduler.conf
I0630 19:27:22.682648 144400 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/admin.conf
I0630 19:27:22.689476 144400 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/kubelet.conf
I0630 19:27:22.695844 144400 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/controller-manager.conf
I0630 19:27:22.702235 144400 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/scheduler.conf
I0630 19:27:22.708395 144400 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0630 19:27:22.714868 144400 kubeadm.go:703] reconfiguring cluster from /var/tmp/minikube/kubeadm.yaml
I0630 19:27:22.714885 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init phase certs all --config /var/tmp/minikube/kubeadm.yaml"
I0630 19:27:22.762797 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init phase kubeconfig all --config /var/tmp/minikube/kubeadm.yaml"
I0630 19:27:23.574870 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init phase kubelet-start --config /var/tmp/minikube/kubeadm.yaml"
I0630 19:27:23.773871 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init phase control-plane all --config /var/tmp/minikube/kubeadm.yaml"
I0630 19:27:23.831583 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init phase etcd local --config /var/tmp/minikube/kubeadm.yaml"
I0630 19:27:23.874299 144400 api_server.go:51] waiting for apiserver process to appear ...
I0630 19:27:23.874361 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:24.384143 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:24.884465 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:25.384491 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:25.894605 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:26.384162 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:26.884303 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:27.383603 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:27.884583 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:28.384160 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:28.884052 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:29.383627 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:29.884578 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:30.384536 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:30.884159 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:31.383643 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:31.884503 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:32.384188 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:32.883570 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:33.384024 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:33.884156 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:34.384435 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:34.884052 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:35.384007 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:35.883866 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:36.384378 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:36.884503 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:37.384396 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:37.884556 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:38.383820 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:38.883630 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:39.384461 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:39.883993 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:40.383684 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:40.884405 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:41.384546 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:41.884578 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:42.383753 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:42.884271 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:43.383912 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:43.884586 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:44.384441 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:44.883584 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:45.384472 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:45.884484 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:46.383561 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:46.883720 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:47.384496 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:47.884566 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:48.383629 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:48.884345 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:49.384553 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:49.884025 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:50.384275 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:50.883648 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:51.384298 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:51.883869 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:52.383586 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:52.883647 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:53.383880 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:53.884434 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:54.384362 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:54.884488 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:55.383631 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:55.884515 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:56.384541 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:56.883543 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:57.384318 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:57.884435 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:58.384603 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:58.883625 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:59.383575 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:27:59.884315 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:00.383837 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:00.883629 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:01.384279 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:01.884293 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:02.384382 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:02.884584 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:03.384356 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:03.884489 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:04.384309 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:04.883940 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:05.384373 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:05.883587 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:06.383862 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:06.883602 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:07.384090 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:07.884639 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:08.383570 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:08.883725 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:09.384169 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:09.884197 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:10.384053 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:10.884312 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:11.384080 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:11.884147 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:12.384399 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:12.884197 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:13.384261 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:13.884550 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:14.384084 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:14.884106 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:15.383629 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:15.883800 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:16.383969 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:16.884372 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:17.383589 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:17.884061 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:18.384028 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:18.884573 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:19.383955 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:19.883610 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:20.384239 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:20.884518 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:21.384270 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:21.883599 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:22.384295 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:22.884051 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:23.383620 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:23.883721 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:28:23.883815 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:28:23.919083 144400 cri.go:87] found id: ""
I0630 19:28:23.919108 144400 logs.go:274] 0 containers: []
W0630 19:28:23.919117 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:28:23.919125 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:28:23.919181 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:28:23.951774 144400 cri.go:87] found id: ""
I0630 19:28:23.951801 144400 logs.go:274] 0 containers: []
W0630 19:28:23.951810 144400 logs.go:276] No container was found matching "etcd"
I0630 19:28:23.951819 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:28:23.951872 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:28:23.992062 144400 cri.go:87] found id: ""
I0630 19:28:23.992080 144400 logs.go:274] 0 containers: []
W0630 19:28:23.992088 144400 logs.go:276] No container was found matching "coredns"
I0630 19:28:23.992096 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:28:23.992160 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:28:24.028664 144400 cri.go:87] found id: ""
I0630 19:28:24.028689 144400 logs.go:274] 0 containers: []
W0630 19:28:24.028702 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:28:24.028711 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:28:24.028763 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:28:24.078124 144400 cri.go:87] found id: ""
I0630 19:28:24.078150 144400 logs.go:274] 0 containers: []
W0630 19:28:24.078158 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:28:24.078165 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:28:24.078228 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:28:24.133612 144400 cri.go:87] found id: ""
I0630 19:28:24.133641 144400 logs.go:274] 0 containers: []
W0630 19:28:24.133650 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:28:24.133660 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:28:24.133708 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:28:24.173968 144400 cri.go:87] found id: ""
I0630 19:28:24.173995 144400 logs.go:274] 0 containers: []
W0630 19:28:24.174004 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:28:24.174012 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:28:24.174065 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:28:24.210596 144400 cri.go:87] found id: ""
I0630 19:28:24.210626 144400 logs.go:274] 0 containers: []
W0630 19:28:24.210642 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:28:24.210656 144400 logs.go:123] Gathering logs for container status ...
I0630 19:28:24.210674 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:28:24.247522 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:28:24.247558 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:28:24.317818 144400 logs.go:138] Found kubelet problem: Jun 30 19:28:24 kubernetes-upgrade-20220630192547-10455 kubelet[2387]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:28:24.390305 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:28:24.390345 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:28:24.407451 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:28:24.407474 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:28:24.476335 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:28:24.476362 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:28:24.476375 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:28:24.525945 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:28:24.525981 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:28:24.526154 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:28:24.526172 144400 out.go:239] Jun 30 19:28:24 kubernetes-upgrade-20220630192547-10455 kubelet[2387]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:28:24 kubernetes-upgrade-20220630192547-10455 kubelet[2387]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:28:24.526179 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:28:24.526187 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:28:34.527883 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:34.883599 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:28:34.883671 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:28:34.918877 144400 cri.go:87] found id: ""
I0630 19:28:34.918900 144400 logs.go:274] 0 containers: []
W0630 19:28:34.918912 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:28:34.918919 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:28:34.918965 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:28:34.955093 144400 cri.go:87] found id: ""
I0630 19:28:34.955115 144400 logs.go:274] 0 containers: []
W0630 19:28:34.955121 144400 logs.go:276] No container was found matching "etcd"
I0630 19:28:34.955126 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:28:34.955169 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:28:34.983099 144400 cri.go:87] found id: ""
I0630 19:28:34.983127 144400 logs.go:274] 0 containers: []
W0630 19:28:34.983137 144400 logs.go:276] No container was found matching "coredns"
I0630 19:28:34.983144 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:28:34.983202 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:28:35.011877 144400 cri.go:87] found id: ""
I0630 19:28:35.011903 144400 logs.go:274] 0 containers: []
W0630 19:28:35.011911 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:28:35.011919 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:28:35.011972 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:28:35.036984 144400 cri.go:87] found id: ""
I0630 19:28:35.037008 144400 logs.go:274] 0 containers: []
W0630 19:28:35.037015 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:28:35.037022 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:28:35.037072 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:28:35.067538 144400 cri.go:87] found id: ""
I0630 19:28:35.067566 144400 logs.go:274] 0 containers: []
W0630 19:28:35.067574 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:28:35.067581 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:28:35.067633 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:28:35.096047 144400 cri.go:87] found id: ""
I0630 19:28:35.096074 144400 logs.go:274] 0 containers: []
W0630 19:28:35.096083 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:28:35.096090 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:28:35.096141 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:28:35.126490 144400 cri.go:87] found id: ""
I0630 19:28:35.126516 144400 logs.go:274] 0 containers: []
W0630 19:28:35.126530 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:28:35.126541 144400 logs.go:123] Gathering logs for container status ...
I0630 19:28:35.126557 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:28:35.155842 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:28:35.155874 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:28:35.225967 144400 logs.go:138] Found kubelet problem: Jun 30 19:28:34 kubernetes-upgrade-20220630192547-10455 kubelet[2597]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:28:35.293568 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:28:35.293610 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:28:35.308822 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:28:35.308851 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:28:35.373560 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:28:35.373592 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:28:35.373606 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:28:35.424013 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:28:35.424049 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:28:35.424210 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:28:35.424226 144400 out.go:239] Jun 30 19:28:34 kubernetes-upgrade-20220630192547-10455 kubelet[2597]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:28:34 kubernetes-upgrade-20220630192547-10455 kubelet[2597]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:28:35.424237 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:28:35.424246 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:28:45.425599 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:45.883563 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:28:45.883642 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:28:45.919679 144400 cri.go:87] found id: ""
I0630 19:28:45.919709 144400 logs.go:274] 0 containers: []
W0630 19:28:45.919720 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:28:45.919728 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:28:45.919780 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:28:45.953700 144400 cri.go:87] found id: ""
I0630 19:28:45.953736 144400 logs.go:274] 0 containers: []
W0630 19:28:45.953746 144400 logs.go:276] No container was found matching "etcd"
I0630 19:28:45.953816 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:28:45.953874 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:28:45.977338 144400 cri.go:87] found id: ""
I0630 19:28:45.977368 144400 logs.go:274] 0 containers: []
W0630 19:28:45.977377 144400 logs.go:276] No container was found matching "coredns"
I0630 19:28:45.977386 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:28:45.977440 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:28:46.001813 144400 cri.go:87] found id: ""
I0630 19:28:46.001835 144400 logs.go:274] 0 containers: []
W0630 19:28:46.001842 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:28:46.001848 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:28:46.001899 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:28:46.027790 144400 cri.go:87] found id: ""
I0630 19:28:46.027812 144400 logs.go:274] 0 containers: []
W0630 19:28:46.027818 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:28:46.027825 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:28:46.027864 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:28:46.056920 144400 cri.go:87] found id: ""
I0630 19:28:46.056942 144400 logs.go:274] 0 containers: []
W0630 19:28:46.056950 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:28:46.056958 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:28:46.057010 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:28:46.083730 144400 cri.go:87] found id: ""
I0630 19:28:46.083756 144400 logs.go:274] 0 containers: []
W0630 19:28:46.083764 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:28:46.083773 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:28:46.083814 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:28:46.110901 144400 cri.go:87] found id: ""
I0630 19:28:46.110920 144400 logs.go:274] 0 containers: []
W0630 19:28:46.110925 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:28:46.110933 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:28:46.110947 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:28:46.163210 144400 logs.go:138] Found kubelet problem: Jun 30 19:28:45 kubernetes-upgrade-20220630192547-10455 kubelet[2889]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:28:46.212262 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:28:46.212294 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:28:46.227562 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:28:46.227587 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:28:46.285029 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:28:46.285053 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:28:46.285066 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:28:46.324472 144400 logs.go:123] Gathering logs for container status ...
I0630 19:28:46.324510 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:28:46.350865 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:28:46.350889 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:28:46.350987 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:28:46.351000 144400 out.go:239] Jun 30 19:28:45 kubernetes-upgrade-20220630192547-10455 kubelet[2889]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:28:45 kubernetes-upgrade-20220630192547-10455 kubelet[2889]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:28:46.351004 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:28:46.351010 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:28:56.351513 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:28:56.383860 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:28:56.383932 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:28:56.411286 144400 cri.go:87] found id: ""
I0630 19:28:56.411312 144400 logs.go:274] 0 containers: []
W0630 19:28:56.411321 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:28:56.411328 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:28:56.411374 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:28:56.434717 144400 cri.go:87] found id: ""
I0630 19:28:56.434740 144400 logs.go:274] 0 containers: []
W0630 19:28:56.434748 144400 logs.go:276] No container was found matching "etcd"
I0630 19:28:56.434755 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:28:56.434800 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:28:56.458857 144400 cri.go:87] found id: ""
I0630 19:28:56.458882 144400 logs.go:274] 0 containers: []
W0630 19:28:56.458891 144400 logs.go:276] No container was found matching "coredns"
I0630 19:28:56.458898 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:28:56.458944 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:28:56.481938 144400 cri.go:87] found id: ""
I0630 19:28:56.481958 144400 logs.go:274] 0 containers: []
W0630 19:28:56.481962 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:28:56.481968 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:28:56.482002 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:28:56.507269 144400 cri.go:87] found id: ""
I0630 19:28:56.507291 144400 logs.go:274] 0 containers: []
W0630 19:28:56.507300 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:28:56.507306 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:28:56.507360 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:28:56.531884 144400 cri.go:87] found id: ""
I0630 19:28:56.531910 144400 logs.go:274] 0 containers: []
W0630 19:28:56.531917 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:28:56.531925 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:28:56.531972 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:28:56.556379 144400 cri.go:87] found id: ""
I0630 19:28:56.556405 144400 logs.go:274] 0 containers: []
W0630 19:28:56.556413 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:28:56.556420 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:28:56.556475 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:28:56.581318 144400 cri.go:87] found id: ""
I0630 19:28:56.581346 144400 logs.go:274] 0 containers: []
W0630 19:28:56.581355 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:28:56.581365 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:28:56.581375 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:28:56.596385 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:28:56.596418 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:28:56.648929 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:28:56.648954 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:28:56.648967 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:28:56.690536 144400 logs.go:123] Gathering logs for container status ...
I0630 19:28:56.690582 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:28:56.721098 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:28:56.721128 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:28:56.773052 144400 logs.go:138] Found kubelet problem: Jun 30 19:28:56 kubernetes-upgrade-20220630192547-10455 kubelet[3180]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:28:56.822309 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:28:56.822339 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:28:56.822458 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:28:56.822477 144400 out.go:239] Jun 30 19:28:56 kubernetes-upgrade-20220630192547-10455 kubelet[3180]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:28:56 kubernetes-upgrade-20220630192547-10455 kubelet[3180]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:28:56.822484 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:28:56.822498 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:29:06.824080 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:29:06.883795 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:29:06.883865 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:29:06.909580 144400 cri.go:87] found id: ""
I0630 19:29:06.909608 144400 logs.go:274] 0 containers: []
W0630 19:29:06.909616 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:29:06.909624 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:29:06.909688 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:29:06.931895 144400 cri.go:87] found id: ""
I0630 19:29:06.931917 144400 logs.go:274] 0 containers: []
W0630 19:29:06.931925 144400 logs.go:276] No container was found matching "etcd"
I0630 19:29:06.931932 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:29:06.931986 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:29:06.954787 144400 cri.go:87] found id: ""
I0630 19:29:06.954811 144400 logs.go:274] 0 containers: []
W0630 19:29:06.954819 144400 logs.go:276] No container was found matching "coredns"
I0630 19:29:06.954826 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:29:06.954875 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:29:06.977415 144400 cri.go:87] found id: ""
I0630 19:29:06.977438 144400 logs.go:274] 0 containers: []
W0630 19:29:06.977446 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:29:06.977453 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:29:06.977503 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:29:07.011543 144400 cri.go:87] found id: ""
I0630 19:29:07.011572 144400 logs.go:274] 0 containers: []
W0630 19:29:07.011581 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:29:07.011589 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:29:07.011647 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:29:07.036655 144400 cri.go:87] found id: ""
I0630 19:29:07.036681 144400 logs.go:274] 0 containers: []
W0630 19:29:07.036688 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:29:07.036694 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:29:07.036737 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:29:07.068895 144400 cri.go:87] found id: ""
I0630 19:29:07.068921 144400 logs.go:274] 0 containers: []
W0630 19:29:07.068929 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:29:07.068937 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:29:07.068986 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:29:07.097631 144400 cri.go:87] found id: ""
I0630 19:29:07.097662 144400 logs.go:274] 0 containers: []
W0630 19:29:07.097671 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:29:07.097683 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:29:07.097699 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:29:07.153974 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:29:07.154003 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:29:07.154017 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:29:07.198026 144400 logs.go:123] Gathering logs for container status ...
I0630 19:29:07.198060 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:29:07.224245 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:29:07.224282 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:29:07.294834 144400 logs.go:138] Found kubelet problem: Jun 30 19:29:06 kubernetes-upgrade-20220630192547-10455 kubelet[3472]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:29:07.357422 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:29:07.357463 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:29:07.377855 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:29:07.377886 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:29:07.378430 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:29:07.378459 144400 out.go:239] Jun 30 19:29:06 kubernetes-upgrade-20220630192547-10455 kubelet[3472]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:29:06 kubernetes-upgrade-20220630192547-10455 kubelet[3472]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:29:07.378467 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:29:07.378477 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:29:17.380320 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:29:17.883996 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:29:17.884059 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:29:17.908195 144400 cri.go:87] found id: ""
I0630 19:29:17.908221 144400 logs.go:274] 0 containers: []
W0630 19:29:17.908229 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:29:17.908236 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:29:17.908291 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:29:17.930591 144400 cri.go:87] found id: ""
I0630 19:29:17.930612 144400 logs.go:274] 0 containers: []
W0630 19:29:17.930617 144400 logs.go:276] No container was found matching "etcd"
I0630 19:29:17.930623 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:29:17.930685 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:29:17.951916 144400 cri.go:87] found id: ""
I0630 19:29:17.951941 144400 logs.go:274] 0 containers: []
W0630 19:29:17.951947 144400 logs.go:276] No container was found matching "coredns"
I0630 19:29:17.951954 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:29:17.952003 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:29:17.974134 144400 cri.go:87] found id: ""
I0630 19:29:17.974158 144400 logs.go:274] 0 containers: []
W0630 19:29:17.974167 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:29:17.974174 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:29:17.974230 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:29:17.996211 144400 cri.go:87] found id: ""
I0630 19:29:17.996236 144400 logs.go:274] 0 containers: []
W0630 19:29:17.996242 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:29:17.996248 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:29:17.996295 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:29:18.019939 144400 cri.go:87] found id: ""
I0630 19:29:18.019961 144400 logs.go:274] 0 containers: []
W0630 19:29:18.019967 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:29:18.019974 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:29:18.020025 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:29:18.043873 144400 cri.go:87] found id: ""
I0630 19:29:18.043899 144400 logs.go:274] 0 containers: []
W0630 19:29:18.043907 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:29:18.043915 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:29:18.043972 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:29:18.067927 144400 cri.go:87] found id: ""
I0630 19:29:18.067950 144400 logs.go:274] 0 containers: []
W0630 19:29:18.067957 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:29:18.067966 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:29:18.067978 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:29:18.117107 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:29:18.117137 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:29:18.117151 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:29:18.158355 144400 logs.go:123] Gathering logs for container status ...
I0630 19:29:18.158391 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:29:18.184435 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:29:18.184469 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:29:18.229656 144400 logs.go:138] Found kubelet problem: Jun 30 19:29:18 kubernetes-upgrade-20220630192547-10455 kubelet[3886]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:29:18.274270 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:29:18.274300 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:29:18.290185 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:29:18.290208 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:29:18.290304 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:29:18.290317 144400 out.go:239] Jun 30 19:29:18 kubernetes-upgrade-20220630192547-10455 kubelet[3886]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:29:18 kubernetes-upgrade-20220630192547-10455 kubelet[3886]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:29:18.290321 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:29:18.290326 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:29:28.290896 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:29:28.383844 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:29:28.383921 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:29:28.414037 144400 cri.go:87] found id: ""
I0630 19:29:28.414065 144400 logs.go:274] 0 containers: []
W0630 19:29:28.414073 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:29:28.414082 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:29:28.414137 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:29:28.443819 144400 cri.go:87] found id: ""
I0630 19:29:28.443853 144400 logs.go:274] 0 containers: []
W0630 19:29:28.443863 144400 logs.go:276] No container was found matching "etcd"
I0630 19:29:28.443871 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:29:28.443926 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:29:28.474311 144400 cri.go:87] found id: ""
I0630 19:29:28.474341 144400 logs.go:274] 0 containers: []
W0630 19:29:28.474349 144400 logs.go:276] No container was found matching "coredns"
I0630 19:29:28.474357 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:29:28.474408 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:29:28.510807 144400 cri.go:87] found id: ""
I0630 19:29:28.510848 144400 logs.go:274] 0 containers: []
W0630 19:29:28.510856 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:29:28.510864 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:29:28.510916 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:29:28.572119 144400 cri.go:87] found id: ""
I0630 19:29:28.572145 144400 logs.go:274] 0 containers: []
W0630 19:29:28.572153 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:29:28.572160 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:29:28.572211 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:29:28.604924 144400 cri.go:87] found id: ""
I0630 19:29:28.604952 144400 logs.go:274] 0 containers: []
W0630 19:29:28.604965 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:29:28.604972 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:29:28.605025 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:29:28.682024 144400 cri.go:87] found id: ""
I0630 19:29:28.682048 144400 logs.go:274] 0 containers: []
W0630 19:29:28.682054 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:29:28.682060 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:29:28.682107 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:29:28.707235 144400 cri.go:87] found id: ""
I0630 19:29:28.707257 144400 logs.go:274] 0 containers: []
W0630 19:29:28.707264 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:29:28.707276 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:29:28.707296 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:29:28.763923 144400 logs.go:138] Found kubelet problem: Jun 30 19:29:28 kubernetes-upgrade-20220630192547-10455 kubelet[4141]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:29:28.820681 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:29:28.820718 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:29:28.837680 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:29:28.837709 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:29:28.904696 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:29:28.904721 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:29:28.904732 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:29:28.948125 144400 logs.go:123] Gathering logs for container status ...
I0630 19:29:28.948156 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:29:28.976510 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:29:28.976539 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:29:28.976683 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:29:28.976706 144400 out.go:239] Jun 30 19:29:28 kubernetes-upgrade-20220630192547-10455 kubelet[4141]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:29:28 kubernetes-upgrade-20220630192547-10455 kubelet[4141]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:29:28.976718 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:29:28.976734 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:29:38.978739 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:29:39.383897 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:29:39.383968 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:29:39.407166 144400 cri.go:87] found id: ""
I0630 19:29:39.407195 144400 logs.go:274] 0 containers: []
W0630 19:29:39.407205 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:29:39.407220 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:29:39.407263 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:29:39.429624 144400 cri.go:87] found id: ""
I0630 19:29:39.429650 144400 logs.go:274] 0 containers: []
W0630 19:29:39.429658 144400 logs.go:276] No container was found matching "etcd"
I0630 19:29:39.429665 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:29:39.429717 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:29:39.453242 144400 cri.go:87] found id: ""
I0630 19:29:39.453266 144400 logs.go:274] 0 containers: []
W0630 19:29:39.453274 144400 logs.go:276] No container was found matching "coredns"
I0630 19:29:39.453283 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:29:39.453333 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:29:39.475246 144400 cri.go:87] found id: ""
I0630 19:29:39.475274 144400 logs.go:274] 0 containers: []
W0630 19:29:39.475282 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:29:39.475290 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:29:39.475332 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:29:39.496888 144400 cri.go:87] found id: ""
I0630 19:29:39.496912 144400 logs.go:274] 0 containers: []
W0630 19:29:39.496918 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:29:39.496924 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:29:39.496961 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:29:39.519465 144400 cri.go:87] found id: ""
I0630 19:29:39.519487 144400 logs.go:274] 0 containers: []
W0630 19:29:39.519492 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:29:39.519499 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:29:39.519546 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:29:39.541464 144400 cri.go:87] found id: ""
I0630 19:29:39.541486 144400 logs.go:274] 0 containers: []
W0630 19:29:39.541492 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:29:39.541498 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:29:39.541535 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:29:39.563692 144400 cri.go:87] found id: ""
I0630 19:29:39.563716 144400 logs.go:274] 0 containers: []
W0630 19:29:39.563723 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:29:39.563733 144400 logs.go:123] Gathering logs for container status ...
I0630 19:29:39.563748 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:29:39.591619 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:29:39.591645 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:29:39.635655 144400 logs.go:138] Found kubelet problem: Jun 30 19:29:39 kubernetes-upgrade-20220630192547-10455 kubelet[4372]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:29:39.680658 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:29:39.812271 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:29:39.828462 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:29:39.828492 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:29:39.884221 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:29:39.884245 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:29:39.884257 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:29:39.922593 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:29:39.922624 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:29:39.922739 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:29:39.922753 144400 out.go:239] Jun 30 19:29:39 kubernetes-upgrade-20220630192547-10455 kubelet[4372]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:29:39 kubernetes-upgrade-20220630192547-10455 kubelet[4372]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:29:39.922760 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:29:39.922767 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:29:49.924693 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:29:50.384327 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:29:50.384405 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:29:50.415723 144400 cri.go:87] found id: ""
I0630 19:29:50.415753 144400 logs.go:274] 0 containers: []
W0630 19:29:50.415762 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:29:50.415769 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:29:50.415821 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:29:50.443881 144400 cri.go:87] found id: ""
I0630 19:29:50.443905 144400 logs.go:274] 0 containers: []
W0630 19:29:50.443911 144400 logs.go:276] No container was found matching "etcd"
I0630 19:29:50.443917 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:29:50.443974 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:29:50.468371 144400 cri.go:87] found id: ""
I0630 19:29:50.468400 144400 logs.go:274] 0 containers: []
W0630 19:29:50.468410 144400 logs.go:276] No container was found matching "coredns"
I0630 19:29:50.468418 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:29:50.468465 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:29:50.491985 144400 cri.go:87] found id: ""
I0630 19:29:50.492012 144400 logs.go:274] 0 containers: []
W0630 19:29:50.492021 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:29:50.492029 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:29:50.492089 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:29:50.514793 144400 cri.go:87] found id: ""
I0630 19:29:50.514825 144400 logs.go:274] 0 containers: []
W0630 19:29:50.514834 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:29:50.514841 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:29:50.514894 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:29:50.539987 144400 cri.go:87] found id: ""
I0630 19:29:50.540015 144400 logs.go:274] 0 containers: []
W0630 19:29:50.540024 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:29:50.540035 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:29:50.540091 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:29:50.565030 144400 cri.go:87] found id: ""
I0630 19:29:50.565060 144400 logs.go:274] 0 containers: []
W0630 19:29:50.565070 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:29:50.565077 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:29:50.565134 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:29:50.589304 144400 cri.go:87] found id: ""
I0630 19:29:50.589334 144400 logs.go:274] 0 containers: []
W0630 19:29:50.589343 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:29:50.589355 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:29:50.589367 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:29:50.625428 144400 logs.go:123] Gathering logs for container status ...
I0630 19:29:50.625465 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:29:50.663982 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:29:50.664016 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:29:50.709884 144400 logs.go:138] Found kubelet problem: Jun 30 19:29:50 kubernetes-upgrade-20220630192547-10455 kubelet[4672]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:29:50.766294 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:29:50.766326 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:29:50.780740 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:29:50.780765 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:29:50.836312 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:29:50.836341 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:29:50.836352 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:29:50.836458 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:29:50.836471 144400 out.go:239] Jun 30 19:29:50 kubernetes-upgrade-20220630192547-10455 kubelet[4672]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:29:50 kubernetes-upgrade-20220630192547-10455 kubelet[4672]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:29:50.836475 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:29:50.836480 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:30:00.837843 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:30:00.884170 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:30:00.884238 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:30:00.920181 144400 cri.go:87] found id: ""
I0630 19:30:00.920205 144400 logs.go:274] 0 containers: []
W0630 19:30:00.920218 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:30:00.920226 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:30:00.920278 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:30:00.947468 144400 cri.go:87] found id: ""
I0630 19:30:00.947493 144400 logs.go:274] 0 containers: []
W0630 19:30:00.947500 144400 logs.go:276] No container was found matching "etcd"
I0630 19:30:00.947508 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:30:00.947569 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:30:00.979620 144400 cri.go:87] found id: ""
I0630 19:30:00.979645 144400 logs.go:274] 0 containers: []
W0630 19:30:00.979651 144400 logs.go:276] No container was found matching "coredns"
I0630 19:30:00.979657 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:30:00.979709 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:30:01.005032 144400 cri.go:87] found id: ""
I0630 19:30:01.005059 144400 logs.go:274] 0 containers: []
W0630 19:30:01.005066 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:30:01.005072 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:30:01.005120 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:30:01.031679 144400 cri.go:87] found id: ""
I0630 19:30:01.031705 144400 logs.go:274] 0 containers: []
W0630 19:30:01.031713 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:30:01.031721 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:30:01.031784 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:30:01.059886 144400 cri.go:87] found id: ""
I0630 19:30:01.059908 144400 logs.go:274] 0 containers: []
W0630 19:30:01.059916 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:30:01.059923 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:30:01.059980 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:30:01.087793 144400 cri.go:87] found id: ""
I0630 19:30:01.087825 144400 logs.go:274] 0 containers: []
W0630 19:30:01.087834 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:30:01.087843 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:30:01.087894 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:30:01.112573 144400 cri.go:87] found id: ""
I0630 19:30:01.112602 144400 logs.go:274] 0 containers: []
W0630 19:30:01.112610 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:30:01.112622 144400 logs.go:123] Gathering logs for container status ...
I0630 19:30:01.112675 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:30:01.149793 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:30:01.149827 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:30:01.213488 144400 logs.go:138] Found kubelet problem: Jun 30 19:30:00 kubernetes-upgrade-20220630192547-10455 kubelet[4961]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:30:01.282343 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:30:01.282384 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:30:01.297738 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:30:01.297793 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:30:01.359784 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:30:01.359813 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:30:01.359825 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:30:01.400323 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:30:01.400355 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:30:01.400492 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:30:01.400506 144400 out.go:239] Jun 30 19:30:00 kubernetes-upgrade-20220630192547-10455 kubelet[4961]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:30:00 kubernetes-upgrade-20220630192547-10455 kubelet[4961]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:30:01.400511 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:30:01.400517 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:30:11.402473 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:30:11.884220 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:30:11.884306 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:30:11.909995 144400 cri.go:87] found id: ""
I0630 19:30:11.910022 144400 logs.go:274] 0 containers: []
W0630 19:30:11.910030 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:30:11.910038 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:30:11.910089 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:30:11.937381 144400 cri.go:87] found id: ""
I0630 19:30:11.937408 144400 logs.go:274] 0 containers: []
W0630 19:30:11.937416 144400 logs.go:276] No container was found matching "etcd"
I0630 19:30:11.937423 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:30:11.937469 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:30:11.962618 144400 cri.go:87] found id: ""
I0630 19:30:11.962640 144400 logs.go:274] 0 containers: []
W0630 19:30:11.962648 144400 logs.go:276] No container was found matching "coredns"
I0630 19:30:11.962657 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:30:11.962715 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:30:11.987269 144400 cri.go:87] found id: ""
I0630 19:30:11.987293 144400 logs.go:274] 0 containers: []
W0630 19:30:11.987301 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:30:11.987308 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:30:11.987357 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:30:12.011128 144400 cri.go:87] found id: ""
I0630 19:30:12.011154 144400 logs.go:274] 0 containers: []
W0630 19:30:12.011162 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:30:12.011170 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:30:12.011218 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:30:12.039722 144400 cri.go:87] found id: ""
I0630 19:30:12.039742 144400 logs.go:274] 0 containers: []
W0630 19:30:12.039750 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:30:12.039757 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:30:12.039802 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:30:12.063833 144400 cri.go:87] found id: ""
I0630 19:30:12.063859 144400 logs.go:274] 0 containers: []
W0630 19:30:12.063867 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:30:12.063876 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:30:12.063920 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:30:12.087520 144400 cri.go:87] found id: ""
I0630 19:30:12.087545 144400 logs.go:274] 0 containers: []
W0630 19:30:12.087552 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:30:12.087563 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:30:12.087576 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:30:12.154621 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:30:12.154647 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:30:12.154662 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:30:12.201917 144400 logs.go:123] Gathering logs for container status ...
I0630 19:30:12.201949 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:30:12.232729 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:30:12.232755 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:30:12.300181 144400 logs.go:138] Found kubelet problem: Jun 30 19:30:12 kubernetes-upgrade-20220630192547-10455 kubelet[5366]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:30:12.354742 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:30:12.354782 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:30:12.369151 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:30:12.369172 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:30:12.369274 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:30:12.369285 144400 out.go:239] Jun 30 19:30:12 kubernetes-upgrade-20220630192547-10455 kubelet[5366]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:30:12 kubernetes-upgrade-20220630192547-10455 kubelet[5366]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:30:12.369289 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:30:12.369294 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:30:22.371413 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:30:22.384454 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:30:22.384524 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:30:22.411938 144400 cri.go:87] found id: ""
I0630 19:30:22.411967 144400 logs.go:274] 0 containers: []
W0630 19:30:22.411979 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:30:22.411985 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:30:22.412038 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:30:22.442026 144400 cri.go:87] found id: ""
I0630 19:30:22.442059 144400 logs.go:274] 0 containers: []
W0630 19:30:22.442067 144400 logs.go:276] No container was found matching "etcd"
I0630 19:30:22.442075 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:30:22.442127 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:30:22.472992 144400 cri.go:87] found id: ""
I0630 19:30:22.473016 144400 logs.go:274] 0 containers: []
W0630 19:30:22.473022 144400 logs.go:276] No container was found matching "coredns"
I0630 19:30:22.473027 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:30:22.473073 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:30:22.511423 144400 cri.go:87] found id: ""
I0630 19:30:22.511462 144400 logs.go:274] 0 containers: []
W0630 19:30:22.511471 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:30:22.511478 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:30:22.511529 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:30:22.540851 144400 cri.go:87] found id: ""
I0630 19:30:22.540881 144400 logs.go:274] 0 containers: []
W0630 19:30:22.540889 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:30:22.540897 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:30:22.540952 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:30:22.586081 144400 cri.go:87] found id: ""
I0630 19:30:22.586109 144400 logs.go:274] 0 containers: []
W0630 19:30:22.586117 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:30:22.586125 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:30:22.586173 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:30:22.634169 144400 cri.go:87] found id: ""
I0630 19:30:22.634195 144400 logs.go:274] 0 containers: []
W0630 19:30:22.634205 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:30:22.634212 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:30:22.634269 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:30:22.682294 144400 cri.go:87] found id: ""
I0630 19:30:22.682323 144400 logs.go:274] 0 containers: []
W0630 19:30:22.682333 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:30:22.682345 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:30:22.682359 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:30:22.737451 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:30:22.737478 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:30:22.737494 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:30:22.789618 144400 logs.go:123] Gathering logs for container status ...
I0630 19:30:22.789665 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:30:22.817230 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:30:22.817260 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:30:22.876940 144400 logs.go:138] Found kubelet problem: Jun 30 19:30:22 kubernetes-upgrade-20220630192547-10455 kubelet[5628]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:30:22.923605 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:30:22.923639 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:30:22.939736 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:30:22.939763 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:30:22.939872 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:30:22.939892 144400 out.go:239] Jun 30 19:30:22 kubernetes-upgrade-20220630192547-10455 kubelet[5628]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:30:22 kubernetes-upgrade-20220630192547-10455 kubelet[5628]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:30:22.939899 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:30:22.939906 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:30:32.940255 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:30:33.384298 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:30:33.384371 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:30:33.412522 144400 cri.go:87] found id: ""
I0630 19:30:33.412548 144400 logs.go:274] 0 containers: []
W0630 19:30:33.412555 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:30:33.412561 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:30:33.412604 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:30:33.437600 144400 cri.go:87] found id: ""
I0630 19:30:33.437629 144400 logs.go:274] 0 containers: []
W0630 19:30:33.437638 144400 logs.go:276] No container was found matching "etcd"
I0630 19:30:33.437646 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:30:33.437712 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:30:33.465356 144400 cri.go:87] found id: ""
I0630 19:30:33.465375 144400 logs.go:274] 0 containers: []
W0630 19:30:33.465381 144400 logs.go:276] No container was found matching "coredns"
I0630 19:30:33.465386 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:30:33.465437 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:30:33.491436 144400 cri.go:87] found id: ""
I0630 19:30:33.491463 144400 logs.go:274] 0 containers: []
W0630 19:30:33.491471 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:30:33.491478 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:30:33.491532 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:30:33.516290 144400 cri.go:87] found id: ""
I0630 19:30:33.516313 144400 logs.go:274] 0 containers: []
W0630 19:30:33.516320 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:30:33.516327 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:30:33.516376 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:30:33.541064 144400 cri.go:87] found id: ""
I0630 19:30:33.541090 144400 logs.go:274] 0 containers: []
W0630 19:30:33.541099 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:30:33.541107 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:30:33.541163 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:30:33.569343 144400 cri.go:87] found id: ""
I0630 19:30:33.569372 144400 logs.go:274] 0 containers: []
W0630 19:30:33.569381 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:30:33.569389 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:30:33.569443 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:30:33.598713 144400 cri.go:87] found id: ""
I0630 19:30:33.598741 144400 logs.go:274] 0 containers: []
W0630 19:30:33.598749 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:30:33.598761 144400 logs.go:123] Gathering logs for container status ...
I0630 19:30:33.598775 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:30:33.627456 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:30:33.627486 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:30:33.695293 144400 logs.go:138] Found kubelet problem: Jun 30 19:30:33 kubernetes-upgrade-20220630192547-10455 kubelet[5833]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:30:33.754077 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:30:33.754109 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:30:33.770398 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:30:33.770424 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:30:33.823560 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:30:33.823612 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:30:33.823634 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:30:33.865105 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:30:33.865204 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:30:33.865394 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:30:33.865458 144400 out.go:239] Jun 30 19:30:33 kubernetes-upgrade-20220630192547-10455 kubelet[5833]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:30:33 kubernetes-upgrade-20220630192547-10455 kubelet[5833]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:30:33.865490 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:30:33.865525 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:30:43.866889 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:30:43.883523 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:30:43.883583 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:30:43.906049 144400 cri.go:87] found id: ""
I0630 19:30:43.906073 144400 logs.go:274] 0 containers: []
W0630 19:30:43.906081 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:30:43.906088 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:30:43.906138 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:30:43.928787 144400 cri.go:87] found id: ""
I0630 19:30:43.928815 144400 logs.go:274] 0 containers: []
W0630 19:30:43.928824 144400 logs.go:276] No container was found matching "etcd"
I0630 19:30:43.928832 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:30:43.928886 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:30:43.956339 144400 cri.go:87] found id: ""
I0630 19:30:43.956362 144400 logs.go:274] 0 containers: []
W0630 19:30:43.956370 144400 logs.go:276] No container was found matching "coredns"
I0630 19:30:43.956378 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:30:43.956429 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:30:43.981413 144400 cri.go:87] found id: ""
I0630 19:30:43.981432 144400 logs.go:274] 0 containers: []
W0630 19:30:43.981438 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:30:43.981444 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:30:43.981489 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:30:44.003797 144400 cri.go:87] found id: ""
I0630 19:30:44.003820 144400 logs.go:274] 0 containers: []
W0630 19:30:44.003828 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:30:44.003838 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:30:44.003888 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:30:44.032949 144400 cri.go:87] found id: ""
I0630 19:30:44.032984 144400 logs.go:274] 0 containers: []
W0630 19:30:44.032990 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:30:44.032996 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:30:44.033046 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:30:44.056430 144400 cri.go:87] found id: ""
I0630 19:30:44.056456 144400 logs.go:274] 0 containers: []
W0630 19:30:44.056465 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:30:44.056473 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:30:44.056521 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:30:44.079069 144400 cri.go:87] found id: ""
I0630 19:30:44.079095 144400 logs.go:274] 0 containers: []
W0630 19:30:44.079103 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:30:44.079112 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:30:44.079121 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:30:44.129914 144400 logs.go:138] Found kubelet problem: Jun 30 19:30:43 kubernetes-upgrade-20220630192547-10455 kubelet[6118]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:30:44.187035 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:30:44.187068 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:30:44.206014 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:30:44.206048 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:30:44.272252 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:30:44.272277 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:30:44.272290 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:30:44.320689 144400 logs.go:123] Gathering logs for container status ...
I0630 19:30:44.320723 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:30:44.352053 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:30:44.352081 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:30:44.352189 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:30:44.352204 144400 out.go:239] Jun 30 19:30:43 kubernetes-upgrade-20220630192547-10455 kubelet[6118]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:30:43 kubernetes-upgrade-20220630192547-10455 kubelet[6118]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:30:44.352217 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:30:44.352224 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:30:54.352999 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:30:54.383827 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:30:54.383913 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:30:54.407680 144400 cri.go:87] found id: ""
I0630 19:30:54.407713 144400 logs.go:274] 0 containers: []
W0630 19:30:54.407722 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:30:54.407728 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:30:54.407769 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:30:54.433456 144400 cri.go:87] found id: ""
I0630 19:30:54.433485 144400 logs.go:274] 0 containers: []
W0630 19:30:54.433493 144400 logs.go:276] No container was found matching "etcd"
I0630 19:30:54.433500 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:30:54.433555 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:30:54.458416 144400 cri.go:87] found id: ""
I0630 19:30:54.458444 144400 logs.go:274] 0 containers: []
W0630 19:30:54.458453 144400 logs.go:276] No container was found matching "coredns"
I0630 19:30:54.458461 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:30:54.458503 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:30:54.481973 144400 cri.go:87] found id: ""
I0630 19:30:54.481994 144400 logs.go:274] 0 containers: []
W0630 19:30:54.481999 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:30:54.482005 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:30:54.482055 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:30:54.504413 144400 cri.go:87] found id: ""
I0630 19:30:54.504439 144400 logs.go:274] 0 containers: []
W0630 19:30:54.504445 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:30:54.504451 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:30:54.504498 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:30:54.526524 144400 cri.go:87] found id: ""
I0630 19:30:54.526569 144400 logs.go:274] 0 containers: []
W0630 19:30:54.526575 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:30:54.526583 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:30:54.526623 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:30:54.548662 144400 cri.go:87] found id: ""
I0630 19:30:54.548688 144400 logs.go:274] 0 containers: []
W0630 19:30:54.548696 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:30:54.548704 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:30:54.548761 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:30:54.571538 144400 cri.go:87] found id: ""
I0630 19:30:54.571560 144400 logs.go:274] 0 containers: []
W0630 19:30:54.571565 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:30:54.571578 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:30:54.571593 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:30:54.622652 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:30:54.622687 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:30:54.622702 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:30:54.662937 144400 logs.go:123] Gathering logs for container status ...
I0630 19:30:54.662973 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:30:54.689474 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:30:54.689510 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:30:54.740004 144400 logs.go:138] Found kubelet problem: Jun 30 19:30:54 kubernetes-upgrade-20220630192547-10455 kubelet[6409]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:30:54.789548 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:30:54.789582 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:30:54.805886 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:30:54.805914 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:30:54.806032 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:30:54.806046 144400 out.go:239] Jun 30 19:30:54 kubernetes-upgrade-20220630192547-10455 kubelet[6409]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:30:54 kubernetes-upgrade-20220630192547-10455 kubelet[6409]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:30:54.806052 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:30:54.806060 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:31:04.806731 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:31:04.883806 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:31:04.883876 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:31:04.907618 144400 cri.go:87] found id: ""
I0630 19:31:04.907640 144400 logs.go:274] 0 containers: []
W0630 19:31:04.907646 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:31:04.907652 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:31:04.907693 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:31:04.934389 144400 cri.go:87] found id: ""
I0630 19:31:04.934411 144400 logs.go:274] 0 containers: []
W0630 19:31:04.934417 144400 logs.go:276] No container was found matching "etcd"
I0630 19:31:04.934423 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:31:04.934469 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:31:04.958899 144400 cri.go:87] found id: ""
I0630 19:31:04.958927 144400 logs.go:274] 0 containers: []
W0630 19:31:04.958937 144400 logs.go:276] No container was found matching "coredns"
I0630 19:31:04.958945 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:31:04.958990 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:31:04.989376 144400 cri.go:87] found id: ""
I0630 19:31:04.989402 144400 logs.go:274] 0 containers: []
W0630 19:31:04.989410 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:31:04.989418 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:31:04.989473 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:31:05.014674 144400 cri.go:87] found id: ""
I0630 19:31:05.014703 144400 logs.go:274] 0 containers: []
W0630 19:31:05.014713 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:31:05.014721 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:31:05.014778 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:31:05.041240 144400 cri.go:87] found id: ""
I0630 19:31:05.041263 144400 logs.go:274] 0 containers: []
W0630 19:31:05.041271 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:31:05.041279 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:31:05.041328 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:31:05.068203 144400 cri.go:87] found id: ""
I0630 19:31:05.068241 144400 logs.go:274] 0 containers: []
W0630 19:31:05.068249 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:31:05.068257 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:31:05.068323 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:31:05.096205 144400 cri.go:87] found id: ""
I0630 19:31:05.096234 144400 logs.go:274] 0 containers: []
W0630 19:31:05.096243 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:31:05.096255 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:31:05.096269 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:31:05.110718 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:31:05.110745 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:31:05.159821 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:31:05.159843 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:31:05.159853 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:31:05.196142 144400 logs.go:123] Gathering logs for container status ...
I0630 19:31:05.196175 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:31:05.222093 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:31:05.222134 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:31:05.268709 144400 logs.go:138] Found kubelet problem: Jun 30 19:31:04 kubernetes-upgrade-20220630192547-10455 kubelet[6698]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:31:05.314171 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:31:05.314200 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:31:05.314359 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:31:05.314377 144400 out.go:239] Jun 30 19:31:04 kubernetes-upgrade-20220630192547-10455 kubelet[6698]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:31:04 kubernetes-upgrade-20220630192547-10455 kubelet[6698]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:31:05.314383 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:31:05.314395 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:31:15.314718 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:31:15.383921 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:31:15.383982 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:31:15.407299 144400 cri.go:87] found id: ""
I0630 19:31:15.407320 144400 logs.go:274] 0 containers: []
W0630 19:31:15.407326 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:31:15.407334 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:31:15.407377 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:31:15.432262 144400 cri.go:87] found id: ""
I0630 19:31:15.432285 144400 logs.go:274] 0 containers: []
W0630 19:31:15.432292 144400 logs.go:276] No container was found matching "etcd"
I0630 19:31:15.432297 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:31:15.432337 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:31:15.457194 144400 cri.go:87] found id: ""
I0630 19:31:15.457215 144400 logs.go:274] 0 containers: []
W0630 19:31:15.457220 144400 logs.go:276] No container was found matching "coredns"
I0630 19:31:15.457226 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:31:15.457263 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:31:15.480100 144400 cri.go:87] found id: ""
I0630 19:31:15.480120 144400 logs.go:274] 0 containers: []
W0630 19:31:15.480126 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:31:15.480132 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:31:15.480182 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:31:15.502529 144400 cri.go:87] found id: ""
I0630 19:31:15.502552 144400 logs.go:274] 0 containers: []
W0630 19:31:15.502559 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:31:15.502565 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:31:15.502613 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:31:15.524832 144400 cri.go:87] found id: ""
I0630 19:31:15.524854 144400 logs.go:274] 0 containers: []
W0630 19:31:15.524860 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:31:15.524865 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:31:15.524910 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:31:15.546926 144400 cri.go:87] found id: ""
I0630 19:31:15.546949 144400 logs.go:274] 0 containers: []
W0630 19:31:15.546955 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:31:15.546963 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:31:15.547014 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:31:15.571078 144400 cri.go:87] found id: ""
I0630 19:31:15.571108 144400 logs.go:274] 0 containers: []
W0630 19:31:15.571122 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:31:15.571133 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:31:15.571146 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:31:15.620113 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:31:15.620137 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:31:15.620148 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:31:15.657559 144400 logs.go:123] Gathering logs for container status ...
I0630 19:31:15.657590 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0630 19:31:15.683196 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:31:15.683233 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:31:15.728691 144400 logs.go:138] Found kubelet problem: Jun 30 19:31:15 kubernetes-upgrade-20220630192547-10455 kubelet[6992]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:31:15.774411 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:31:15.774444 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:31:15.790725 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:31:15.790751 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
W0630 19:31:15.790858 144400 out.go:239] X Problems detected in kubelet:
X Problems detected in kubelet:
W0630 19:31:15.790873 144400 out.go:239] Jun 30 19:31:15 kubernetes-upgrade-20220630192547-10455 kubelet[6992]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
Jun 30 19:31:15 kubernetes-upgrade-20220630192547-10455 kubelet[6992]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:31:15.790879 144400 out.go:309] Setting ErrFile to fd 2...
I0630 19:31:15.790888 144400 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:31:25.791787 144400 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:31:25.800880 144400 kubeadm.go:630] restartCluster took 4m3.183356513s
W0630 19:31:25.801005 144400 out.go:239] ! Unable to restart cluster, will reset it: apiserver healthz: apiserver process never appeared
! Unable to restart cluster, will reset it: apiserver healthz: apiserver process never appeared
I0630 19:31:25.801029 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm reset --cri-socket /run/containerd/containerd.sock --force"
I0630 19:31:26.518099 144400 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service kubelet
I0630 19:31:26.528747 144400 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0630 19:31:26.539433 144400 kubeadm.go:221] ignoring SystemVerification for kubeadm because of docker driver
I0630 19:31:26.539497 144400 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0630 19:31:26.549353 144400 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0630 19:31:26.549404 144400 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0630 19:31:26.849268 144400 out.go:204] - Generating certificates and keys ...
I0630 19:31:27.968210 144400 out.go:204] - Booting up control plane ...
W0630 19:33:22.984256 144400 out.go:239] ! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1012-gcp[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[0;37mCGROUPS_PIDS[0m: [0;32menabled[0m
[0;37mCGROUPS_HUGETLB[0m: [0;32menabled[0m
[0;37mCGROUPS_BLKIO[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'
stderr:
W0630 19:31:26.594442 7530 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1012-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1012-gcp[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[0;37mCGROUPS_PIDS[0m: [0;32menabled[0m
[0;37mCGROUPS_HUGETLB[0m: [0;32menabled[0m
[0;37mCGROUPS_BLKIO[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'
stderr:
W0630 19:31:26.594442 7530 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1012-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
I0630 19:33:22.984319 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm reset --cri-socket /run/containerd/containerd.sock --force"
I0630 19:33:23.727771 144400 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service kubelet
I0630 19:33:23.739514 144400 kubeadm.go:221] ignoring SystemVerification for kubeadm because of docker driver
I0630 19:33:23.739579 144400 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0630 19:33:23.747240 144400 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0630 19:33:23.747290 144400 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0630 19:33:24.137422 144400 out.go:204] - Generating certificates and keys ...
I0630 19:33:25.331174 144400 out.go:204] - Booting up control plane ...
I0630 19:35:20.321935 144400 kubeadm.go:397] StartCluster complete in 7m57.734528865s
I0630 19:35:20.321981 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:35:20.322027 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:35:20.345506 144400 cri.go:87] found id: ""
I0630 19:35:20.345534 144400 logs.go:274] 0 containers: []
W0630 19:35:20.345542 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:35:20.345550 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:35:20.345601 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:35:20.367499 144400 cri.go:87] found id: ""
I0630 19:35:20.367525 144400 logs.go:274] 0 containers: []
W0630 19:35:20.367531 144400 logs.go:276] No container was found matching "etcd"
I0630 19:35:20.367537 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:35:20.367575 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:35:20.389868 144400 cri.go:87] found id: ""
I0630 19:35:20.389890 144400 logs.go:274] 0 containers: []
W0630 19:35:20.389896 144400 logs.go:276] No container was found matching "coredns"
I0630 19:35:20.389901 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:35:20.389940 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:35:20.412692 144400 cri.go:87] found id: ""
I0630 19:35:20.412714 144400 logs.go:274] 0 containers: []
W0630 19:35:20.412719 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:35:20.412725 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:35:20.412771 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:35:20.434617 144400 cri.go:87] found id: ""
I0630 19:35:20.434644 144400 logs.go:274] 0 containers: []
W0630 19:35:20.434652 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:35:20.434659 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:35:20.434701 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:35:20.459283 144400 cri.go:87] found id: ""
I0630 19:35:20.459304 144400 logs.go:274] 0 containers: []
W0630 19:35:20.459310 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:35:20.459315 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:35:20.459352 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:35:20.484451 144400 cri.go:87] found id: ""
I0630 19:35:20.484473 144400 logs.go:274] 0 containers: []
W0630 19:35:20.484479 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:35:20.484486 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:35:20.484535 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:35:20.507241 144400 cri.go:87] found id: ""
I0630 19:35:20.507260 144400 logs.go:274] 0 containers: []
W0630 19:35:20.507268 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:35:20.507277 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:35:20.507286 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:35:20.556138 144400 logs.go:138] Found kubelet problem: Jun 30 19:35:20 kubernetes-upgrade-20220630192547-10455 kubelet[11511]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:35:20.601617 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:35:20.601648 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:35:20.617961 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:35:20.617992 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:35:20.665366 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:35:20.665388 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:35:20.665402 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:35:20.708623 144400 logs.go:123] Gathering logs for container status ...
I0630 19:35:20.708655 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
W0630 19:35:20.735172 144400 out.go:369] Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1012-gcp[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[0;37mCGROUPS_PIDS[0m: [0;32menabled[0m
[0;37mCGROUPS_HUGETLB[0m: [0;32menabled[0m
[0;37mCGROUPS_BLKIO[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'
stderr:
W0630 19:33:23.788059 9651 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1012-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0630 19:35:20.735211 144400 out.go:239] *
*
W0630 19:35:20.735427 144400 out.go:239] X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1012-gcp[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[0;37mCGROUPS_PIDS[0m: [0;32menabled[0m
[0;37mCGROUPS_HUGETLB[0m: [0;32menabled[0m
[0;37mCGROUPS_BLKIO[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'
stderr:
W0630 19:33:23.788059 9651 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1012-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1012-gcp[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[0;37mCGROUPS_PIDS[0m: [0;32menabled[0m
[0;37mCGROUPS_HUGETLB[0m: [0;32menabled[0m
[0;37mCGROUPS_BLKIO[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'
stderr:
W0630 19:33:23.788059 9651 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1012-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0630 19:35:20.735465 144400 out.go:239] *
*
W0630 19:35:20.736207 144400 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0630 19:35:20.738341 144400 out.go:177] X Problems detected in kubelet:
I0630 19:35:20.740371 144400 out.go:177] Jun 30 19:35:20 kubernetes-upgrade-20220630192547-10455 kubelet[11511]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:35:20.743786 144400 out.go:177]
W0630 19:35:20.745382 144400 out.go:239] X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1012-gcp[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[0;37mCGROUPS_PIDS[0m: [0;32menabled[0m
[0;37mCGROUPS_HUGETLB[0m: [0;32menabled[0m
[0;37mCGROUPS_BLKIO[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'
stderr:
W0630 19:33:23.788059 9651 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1012-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1012-gcp[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[0;37mCGROUPS_PIDS[0m: [0;32menabled[0m
[0;37mCGROUPS_HUGETLB[0m: [0;32menabled[0m
[0;37mCGROUPS_BLKIO[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'
stderr:
W0630 19:33:23.788059 9651 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1012-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0630 19:35:20.745524 144400 out.go:239] * Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
* Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
W0630 19:35:20.745593 144400 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/4172
* Related issue: https://github.com/kubernetes/minikube/issues/4172
I0630 19:35:20.747207 144400 out.go:177]
** /stderr **
version_upgrade_test.go:252: failed to upgrade with newest k8s version. args: out/minikube-linux-amd64 start -p kubernetes-upgrade-20220630192547-10455 --memory=2200 --kubernetes-version=v1.24.2 --alsologtostderr -v=1 --driver=docker --container-runtime=containerd : exit status 109
version_upgrade_test.go:255: (dbg) Run: kubectl --context kubernetes-upgrade-20220630192547-10455 version --output=json
version_upgrade_test.go:255: (dbg) Non-zero exit: kubectl --context kubernetes-upgrade-20220630192547-10455 version --output=json: exit status 1 (53.22664ms)
-- stdout --
{
"clientVersion": {
"major": "1",
"minor": "24",
"gitVersion": "v1.24.2",
"gitCommit": "f66044f4361b9f1f96f0053dd46cb7dce5e990a8",
"gitTreeState": "clean",
"buildDate": "2022-06-15T14:22:29Z",
"goVersion": "go1.18.3",
"compiler": "gc",
"platform": "linux/amd64"
},
"kustomizeVersion": "v4.5.4"
}
-- /stdout --
** stderr **
The connection to the server 192.168.67.2:8443 was refused - did you specify the right host or port?
** /stderr **
version_upgrade_test.go:257: error running kubectl: exit status 1
panic.go:482: *** TestKubernetesUpgrade FAILED at 2022-06-30 19:35:20.925095244 +0000 UTC m=+2239.637842907
helpers_test.go:222: -----------------------post-mortem--------------------------------
helpers_test.go:230: ======> post-mortem[TestKubernetesUpgrade]: docker inspect <======
helpers_test.go:231: (dbg) Run: docker inspect kubernetes-upgrade-20220630192547-10455
helpers_test.go:235: (dbg) docker inspect kubernetes-upgrade-20220630192547-10455:
-- stdout --
[
{
"Id": "86a18b04d0ad2292fc276f60e9adad3ee7eb70cca1081fe264c8b36066a383d6",
"Created": "2022-06-30T19:26:09.47594087Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 144740,
"ExitCode": 0,
"Error": "",
"StartedAt": "2022-06-30T19:26:50.541154275Z",
"FinishedAt": "2022-06-30T19:26:48.710313029Z"
},
"Image": "sha256:573e7be5768273a7845baee1ae90fa2e33b83b10a7fbb0f0f41efbf29b53d1f1",
"ResolvConfPath": "/var/lib/docker/containers/86a18b04d0ad2292fc276f60e9adad3ee7eb70cca1081fe264c8b36066a383d6/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/86a18b04d0ad2292fc276f60e9adad3ee7eb70cca1081fe264c8b36066a383d6/hostname",
"HostsPath": "/var/lib/docker/containers/86a18b04d0ad2292fc276f60e9adad3ee7eb70cca1081fe264c8b36066a383d6/hosts",
"LogPath": "/var/lib/docker/containers/86a18b04d0ad2292fc276f60e9adad3ee7eb70cca1081fe264c8b36066a383d6/86a18b04d0ad2292fc276f60e9adad3ee7eb70cca1081fe264c8b36066a383d6-json.log",
"Name": "/kubernetes-upgrade-20220630192547-10455",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "unconfined",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro",
"kubernetes-upgrade-20220630192547-10455:/var"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "kubernetes-upgrade-20220630192547-10455",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "host",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 2306867200,
"NanoCpus": 2000000000,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 4613734400,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/0df8dd0b1e01150b411904d53fc8ed6d85e9d545ae7959b2c3671bcd71bc8fd7-init/diff:/var/lib/docker/overlay2/633bd8009a2ef876fc72811c5f84e0b1eae70cab2d3e943892548c1011e65a81/diff:/var/lib/docker/overlay2/97b28156aebffeac1e36ec4cffc6b227a39fcc34294b71bef52e78be2c75775b/diff:/var/lib/docker/overlay2/ab4a7a456223981edd58a8a2cdafc86acb99de3f10471dbd26e1ec446f1c4293/diff:/var/lib/docker/overlay2/0e8bf0ec03ae0d718998c74558ab9884d1ca55938cdd8e29c2777ebd7e4a5652/diff:/var/lib/docker/overlay2/7c61d9a79ed9eb05ce0a99aef2343904ac5400c884d3900f666147f1f4d123d0/diff:/var/lib/docker/overlay2/948e1c67055a82faa46a5f4d9410ec17b5d4d0bda4183ac2f66761d8718e74cc/diff:/var/lib/docker/overlay2/708ae387ddf8e706ff0ccc800ad9d1bcea12067ee594904e246f2e84a3eb49b3/diff:/var/lib/docker/overlay2/9e9d9efa4deb7b511452993c9fb152bb830feae6494cffc55ea5712c78d66800/diff:/var/lib/docker/overlay2/774ce976f395a7d5738636d898351e15a3d2c1e3f27786491292f7302efab11e/diff:/var/lib/docker/overlay2/0d0b8a
22c38bff5604fc8ad8694946ff7a5d3c487423c7036b3c9f3da26020b3/diff:/var/lib/docker/overlay2/54677001e9f77ef0fa48de89aec93e1ef7e4ccbbf87208434e9d86fb457bacf8/diff:/var/lib/docker/overlay2/11aa4391ae1538dbac05759147d1e581c0742cb0193c5b67b282481b9f3b7234/diff:/var/lib/docker/overlay2/34b923c3df1ca8e50c9153603293164d7056cd21f220ab9d929bbef5caab55b8/diff:/var/lib/docker/overlay2/f0364c40773b5507af04fde9ff8677ee5d0889f80590495a4ea95560094e2805/diff:/var/lib/docker/overlay2/3979c16d414c5d361f5327c01ea288fd40ad19c70864d4fd6a943f85b82961e2/diff:/var/lib/docker/overlay2/a9464d6022e762f54c9ef9b6e8005ccf97626d2c0e37fc69c4b441298f44863c/diff:/var/lib/docker/overlay2/f3608472f5e55bfb446e7d1ddcb1a88069acb61d09ba67c80626e7af159ba3ea/diff:/var/lib/docker/overlay2/0729b64af43bd5795c6a0c3a1367ceb0a2f372ce9fa5b3a376b4dec410302020/diff:/var/lib/docker/overlay2/eb4f1e87d2ad205c8fdfa23f066a95991dcb07f93c491a14df63c0c5b096426b/diff:/var/lib/docker/overlay2/d55b4e43a5260ad88dddccd97bfc18579de781151509896e983db0858fec8e74/diff:/var/lib/d
ocker/overlay2/80460ef0fb87a0d2cbd3b3b2fe6e50642e5b220312f826bd8488317538e69e43/diff:/var/lib/docker/overlay2/3799c10ddb3bb5965310216750bf6c65b3abec3209caed4d6f5da55049998e87/diff:/var/lib/docker/overlay2/d7b5a94eb02933cbfa914ba97dc074400fb2fdef78de5b5072232f9f6ae04339/diff:/var/lib/docker/overlay2/161a3fcaaea1a3a8f58ab0f67da14341a0d98f829a59601fc46853cfbea3cd91/diff:/var/lib/docker/overlay2/8a63082bb84118b4bedb12fdeb1b80d59578faf76ca50b66cbbc3c984941c5ce/diff:/var/lib/docker/overlay2/703ec0bc855ee42991cea682510431d0ccfc231cab33635878cd7b6d371bf6dd/diff:/var/lib/docker/overlay2/29feaa9c5088aae12f4c0efd392f954cb11ab0c570ec2dda590ee941f149f3c0/diff:/var/lib/docker/overlay2/11b21d0e1fee74673648f4ba28f0fd8a24cc000689143a62c488aaec91c786b9/diff:/var/lib/docker/overlay2/8a379a6131952bd8d0d13b43f002e37c2ec4c4f38ad3838dd735d2c3387f5279/diff:/var/lib/docker/overlay2/558154a0ca9cef5593f4bcde2d6ce6f695afb50350d1d78ff9dc3a9317be66c7/diff:/var/lib/docker/overlay2/dda0942a920e4018e6710da272d7adda88018a5077caa14779efc9ba600
5443e/diff:/var/lib/docker/overlay2/c22cee49b9640d89b5a39a8d430d0d9d1adf5151efea231d8823f0eb7aa2b73c/diff:/var/lib/docker/overlay2/1702ce4f51207e86f8f7bc3eb00cd1ef8cea277185d24d06e3bb88936dcc84f9/diff:/var/lib/docker/overlay2/938a173c3080cd68a5de95451dd002ad3b25153ab99c589051b037914208707c/diff:/var/lib/docker/overlay2/aee37c7b17b35d3e47634f1aecf487e2d8e7d17bf6995f2f7f2a93f9630def4b/diff:/var/lib/docker/overlay2/adc4852c929baecec5a3d6b0d9e3e5ad75f1b9ca79fc4e9fb08d78a8e21ddcef/diff:/var/lib/docker/overlay2/514e96d0ae384c02124a0c4d000b44556df7bef6b33cc3247c0c89ef71f750e6/diff:/var/lib/docker/overlay2/8ec2ac5c33c3b5451873404c6539e43b6353d834d1302131071c56f1b84ec7d1/diff:/var/lib/docker/overlay2/937dcee9b8e4727b75f578b09a68e46a221fd59032b98926dc28c8e92d24989b/diff:/var/lib/docker/overlay2/10d855063f11b53ac90e28e320d88c266ca634bd7b4e2dee7b28c719f58d3ee8/diff:/var/lib/docker/overlay2/24fd8feb6f63a1701650abc8940da2972343f652b431cb406532c1647186b0ec/diff",
"MergedDir": "/var/lib/docker/overlay2/0df8dd0b1e01150b411904d53fc8ed6d85e9d545ae7959b2c3671bcd71bc8fd7/merged",
"UpperDir": "/var/lib/docker/overlay2/0df8dd0b1e01150b411904d53fc8ed6d85e9d545ae7959b2c3671bcd71bc8fd7/diff",
"WorkDir": "/var/lib/docker/overlay2/0df8dd0b1e01150b411904d53fc8ed6d85e9d545ae7959b2c3671bcd71bc8fd7/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "volume",
"Name": "kubernetes-upgrade-20220630192547-10455",
"Source": "/var/lib/docker/volumes/kubernetes-upgrade-20220630192547-10455/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
},
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "kubernetes-upgrade-20220630192547-10455",
"Domainname": "",
"User": "root",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"22/tcp": {},
"2376/tcp": {},
"32443/tcp": {},
"5000/tcp": {},
"8443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase-builds:v0.0.32-1656350719-14420@sha256:e7b7f38d1a2eba7828afc2c4c3d24e1d391db431976e47aa6dc5c7a6b038ca4e",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "kubernetes-upgrade-20220630192547-10455",
"name.minikube.sigs.k8s.io": "kubernetes-upgrade-20220630192547-10455",
"role.minikube.sigs.k8s.io": ""
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "9986a0c094e024b5043afbc858c214746ec494aa5e77454038db2c09fc440e38",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "49309"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "49308"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "49305"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "49307"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "49306"
}
]
},
"SandboxKey": "/var/run/docker/netns/9986a0c094e0",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"kubernetes-upgrade-20220630192547-10455": {
"IPAMConfig": {
"IPv4Address": "192.168.67.2"
},
"Links": null,
"Aliases": [
"86a18b04d0ad",
"kubernetes-upgrade-20220630192547-10455"
],
"NetworkID": "bd6759ac35d467ea80fd5cf8f0de00d1b9c0c31954e3c65b82fd013857e15eac",
"EndpointID": "b9890c5797fb65f2ee2d7e70fcbfecf1a83aaa55f61aa0acde0ba0590c9dcf49",
"Gateway": "192.168.67.1",
"IPAddress": "192.168.67.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:c0:a8:43:02",
"DriverOpts": null
}
}
}
}
]
-- /stdout --
helpers_test.go:239: (dbg) Run: out/minikube-linux-amd64 status --format={{.Host}} -p kubernetes-upgrade-20220630192547-10455 -n kubernetes-upgrade-20220630192547-10455
helpers_test.go:239: (dbg) Non-zero exit: out/minikube-linux-amd64 status --format={{.Host}} -p kubernetes-upgrade-20220630192547-10455 -n kubernetes-upgrade-20220630192547-10455: exit status 2 (388.775655ms)
-- stdout --
Running
-- /stdout --
helpers_test.go:239: status error: exit status 2 (may be ok)
helpers_test.go:244: <<< TestKubernetesUpgrade FAILED: start of post-mortem logs <<<
helpers_test.go:245: ======> post-mortem[TestKubernetesUpgrade]: minikube logs <======
helpers_test.go:247: (dbg) Run: out/minikube-linux-amd64 -p kubernetes-upgrade-20220630192547-10455 logs -n 25
helpers_test.go:252: TestKubernetesUpgrade logs:
-- stdout --
*
* ==> Audit <==
* |---------|---------------------------------------------------|----------|---------|---------|---------------------|---------------------|
| Command | Args | Profile | User | Version | Start Time | End Time |
|---------|---------------------------------------------------|----------|---------|---------|---------------------|---------------------|
| stop | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:31 UTC | 30 Jun 22 19:31 UTC |
| | no-preload-20220630193026-10455 | | | | | |
| | --alsologtostderr -v=3 | | | | | |
| addons | enable dashboard -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:31 UTC | 30 Jun 22 19:31 UTC |
| | no-preload-20220630193026-10455 | | | | | |
| | --images=MetricsScraper=k8s.gcr.io/echoserver:1.4 | | | | | |
| start | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:31 UTC | |
| | no-preload-20220630193026-10455 | | | | | |
| | --memory=2200 --alsologtostderr | | | | | |
| | --wait=true --preload=false | | | | | |
| | --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| | --kubernetes-version=v1.24.2 | | | | | |
| addons | enable metrics-server -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:32 UTC | 30 Jun 22 19:32 UTC |
| | old-k8s-version-20220630193000-10455 | | | | | |
| | --images=MetricsServer=k8s.gcr.io/echoserver:1.4 | | | | | |
| | --registries=MetricsServer=fake.domain | | | | | |
| stop | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:32 UTC | 30 Jun 22 19:32 UTC |
| | old-k8s-version-20220630193000-10455 | | | | | |
| | --alsologtostderr -v=3 | | | | | |
| addons | enable dashboard -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:32 UTC | 30 Jun 22 19:32 UTC |
| | old-k8s-version-20220630193000-10455 | | | | | |
| | --images=MetricsScraper=k8s.gcr.io/echoserver:1.4 | | | | | |
| start | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:32 UTC | 30 Jun 22 19:33 UTC |
| | old-k8s-version-20220630193000-10455 | | | | | |
| | --memory=2200 --alsologtostderr | | | | | |
| | --wait=true --kvm-network=default | | | | | |
| | --kvm-qemu-uri=qemu:///system | | | | | |
| | --disable-driver-mounts | | | | | |
| | --keep-context=false --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| | --kubernetes-version=v1.16.0 | | | | | |
| start | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:33 UTC | 30 Jun 22 19:33 UTC |
| | cert-expiration-20220630192935-10455 | | | | | |
| | --memory=2048 | | | | | |
| | --cert-expiration=8760h | | | | | |
| | --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| delete | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:33 UTC | 30 Jun 22 19:33 UTC |
| | cert-expiration-20220630192935-10455 | | | | | |
| start | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:33 UTC | 30 Jun 22 19:34 UTC |
| | embed-certs-20220630193321-10455 | | | | | |
| | --memory=2200 --alsologtostderr | | | | | |
| | --wait=true --embed-certs | | | | | |
| | --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| | --kubernetes-version=v1.24.2 | | | | | |
| ssh | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:33 UTC | 30 Jun 22 19:33 UTC |
| | old-k8s-version-20220630193000-10455 | | | | | |
| | sudo crictl images -o json | | | | | |
| pause | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:33 UTC | 30 Jun 22 19:33 UTC |
| | old-k8s-version-20220630193000-10455 | | | | | |
| | --alsologtostderr -v=1 | | | | | |
| unpause | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:33 UTC | 30 Jun 22 19:33 UTC |
| | old-k8s-version-20220630193000-10455 | | | | | |
| | --alsologtostderr -v=1 | | | | | |
| delete | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:33 UTC | 30 Jun 22 19:33 UTC |
| | old-k8s-version-20220630193000-10455 | | | | | |
| delete | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:33 UTC | 30 Jun 22 19:33 UTC |
| | old-k8s-version-20220630193000-10455 | | | | | |
| delete | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:33 UTC | 30 Jun 22 19:33 UTC |
| | disable-driver-mounts-20220630193353-10455 | | | | | |
| start | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:33 UTC | 30 Jun 22 19:34 UTC |
| | default-k8s-different-port-20220630193354-10455 | | | | | |
| | --memory=2200 --alsologtostderr --wait=true | | | | | |
| | --apiserver-port=8444 --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| | --kubernetes-version=v1.24.2 | | | | | |
| addons | enable metrics-server -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:34 UTC | 30 Jun 22 19:34 UTC |
| | embed-certs-20220630193321-10455 | | | | | |
| | --images=MetricsServer=k8s.gcr.io/echoserver:1.4 | | | | | |
| | --registries=MetricsServer=fake.domain | | | | | |
| stop | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:34 UTC | 30 Jun 22 19:34 UTC |
| | embed-certs-20220630193321-10455 | | | | | |
| | --alsologtostderr -v=3 | | | | | |
| addons | enable metrics-server -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:34 UTC | 30 Jun 22 19:34 UTC |
| | default-k8s-different-port-20220630193354-10455 | | | | | |
| | --images=MetricsServer=k8s.gcr.io/echoserver:1.4 | | | | | |
| | --registries=MetricsServer=fake.domain | | | | | |
| addons | enable dashboard -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:34 UTC | 30 Jun 22 19:34 UTC |
| | embed-certs-20220630193321-10455 | | | | | |
| | --images=MetricsScraper=k8s.gcr.io/echoserver:1.4 | | | | | |
| start | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:34 UTC | |
| | embed-certs-20220630193321-10455 | | | | | |
| | --memory=2200 --alsologtostderr | | | | | |
| | --wait=true --embed-certs | | | | | |
| | --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| | --kubernetes-version=v1.24.2 | | | | | |
| stop | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:34 UTC | 30 Jun 22 19:35 UTC |
| | default-k8s-different-port-20220630193354-10455 | | | | | |
| | --alsologtostderr -v=3 | | | | | |
| addons | enable dashboard -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:35 UTC | 30 Jun 22 19:35 UTC |
| | default-k8s-different-port-20220630193354-10455 | | | | | |
| | --images=MetricsScraper=k8s.gcr.io/echoserver:1.4 | | | | | |
| start | -p | minikube | jenkins | v1.26.0 | 30 Jun 22 19:35 UTC | |
| | default-k8s-different-port-20220630193354-10455 | | | | | |
| | --memory=2200 --alsologtostderr --wait=true | | | | | |
| | --apiserver-port=8444 --driver=docker | | | | | |
| | --container-runtime=containerd | | | | | |
| | --kubernetes-version=v1.24.2 | | | | | |
|---------|---------------------------------------------------|----------|---------|---------|---------------------|---------------------|
*
* ==> Last Start <==
* Log file created at: 2022/06/30 19:35:10
Running on machine: ubuntu-20-agent-10
Binary: Built with gc go1.18.3 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0630 19:35:10.474259 224099 out.go:296] Setting OutFile to fd 1 ...
I0630 19:35:10.474464 224099 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:35:10.474478 224099 out.go:309] Setting ErrFile to fd 2...
I0630 19:35:10.474485 224099 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0630 19:35:10.474908 224099 root.go:329] Updating PATH: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/bin
I0630 19:35:10.475156 224099 out.go:303] Setting JSON to false
I0630 19:35:10.476869 224099 start.go:115] hostinfo: {"hostname":"ubuntu-20-agent-10","uptime":4665,"bootTime":1656613046,"procs":1061,"os":"linux","platform":"ubuntu","platformFamily":"debian","platformVersion":"20.04","kernelVersion":"5.15.0-1012-gcp","kernelArch":"x86_64","virtualizationSystem":"kvm","virtualizationRole":"guest","hostId":"591c9f12-2938-3743-e2bf-c56a050d43d1"}
I0630 19:35:10.476937 224099 start.go:125] virtualization: kvm guest
I0630 19:35:10.479541 224099 out.go:177] * [default-k8s-different-port-20220630193354-10455] minikube v1.26.0 on Ubuntu 20.04 (kvm/amd64)
I0630 19:35:10.481136 224099 out.go:177] - MINIKUBE_LOCATION=14420
I0630 19:35:10.481089 224099 notify.go:193] Checking for updates...
I0630 19:35:10.482814 224099 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0630 19:35:10.484357 224099 out.go:177] - KUBECONFIG=/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/kubeconfig
I0630 19:35:10.485833 224099 out.go:177] - MINIKUBE_HOME=/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube
I0630 19:35:10.487511 224099 out.go:177] - MINIKUBE_BIN=out/minikube-linux-amd64
I0630 19:35:10.489273 224099 config.go:178] Loaded profile config "default-k8s-different-port-20220630193354-10455": Driver=docker, ContainerRuntime=containerd, KubernetesVersion=v1.24.2
I0630 19:35:10.489665 224099 driver.go:360] Setting default libvirt URI to qemu:///system
I0630 19:35:10.531915 224099 docker.go:137] docker version: linux-20.10.17
I0630 19:35:10.532008 224099 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0630 19:35:10.650295 224099 info.go:265] docker info: {ID:TS6T:UINC:MIYS:RZPA:KS6T:4JQK:7JHN:D6RA:LDP2:MHAE:G32M:C5NQ Containers:4 ContainersRunning:3 ContainersPaused:0 ContainersStopped:1 Images:8 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:66 OomKillDisable:true NGoroutines:57 SystemTime:2022-06-30 19:35:10.564037462 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1012-gcp OperatingSystem:Ubuntu 20.04.4 LTS OSType:linux Architecture:x86_64 IndexServ
erAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:8 MemTotal:33662451712 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ubuntu-20-agent-10 Labels:[] ExperimentalBuild:false ServerVersion:20.10.17 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1 Expected:10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1} RuncCommit:{ID:v1.1.2-0-ga916309 Expected:v1.1.2-0-ga916309} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=default] ProductLicense: Warnings:<nil> ServerErrors:[] Clien
tInfo:{Debug:false Plugins:[map[Experimental:true Name:app Path:/usr/libexec/docker/cli-plugins/docker-app SchemaVersion:0.1.0 ShortDescription:Docker App Vendor:Docker Inc. Version:v0.9.1-beta3] map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.8.2-docker] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.6.0] map[Name:scan Path:/usr/libexec/docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.17.0]] Warnings:<nil>}}
I0630 19:35:10.650418 224099 docker.go:254] overlay module found
I0630 19:35:10.652973 224099 out.go:177] * Using the docker driver based on existing profile
I0630 19:35:10.654508 224099 start.go:284] selected driver: docker
I0630 19:35:10.654524 224099 start.go:808] validating driver "docker" against &{Name:default-k8s-different-port-20220630193354-10455 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.32-1656350719-14420@sha256:e7b7f38d1a2eba7828afc2c4c3d24e1d391db431976e47aa6dc5c7a6b038ca4e Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.24.2 ClusterName:default-k8s-different-port
-20220630193354-10455 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8444 NodeName:} Nodes:[{Name: IP:192.168.76.2 Port:8444 KubernetesVersion:v1.24.2 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[dashboard:true default-storageclass:true metrics-server:true storage-provisioner:true] CustomAddonImages:map[Dashboard:kubernetesui/dashboard:v2.6.0@sha256:4af9580485920635d888efe1eddbd67e12f9d5d84dba87100e93feb4e46636b3 MetricsScraper:k8s.gcr.io/echoserver:1.4 MetricsServer:k8s.gcr.io/echoserver:1.4] CustomAddonRegistries:map[MetricsServer:fake.domain] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true syst
em_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath:}
I0630 19:35:10.654628 224099 start.go:819] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0630 19:35:10.655490 224099 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0630 19:35:10.781919 224099 info.go:265] docker info: {ID:TS6T:UINC:MIYS:RZPA:KS6T:4JQK:7JHN:D6RA:LDP2:MHAE:G32M:C5NQ Containers:4 ContainersRunning:3 ContainersPaused:0 ContainersStopped:1 Images:8 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:66 OomKillDisable:true NGoroutines:57 SystemTime:2022-06-30 19:35:10.688317125 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1012-gcp OperatingSystem:Ubuntu 20.04.4 LTS OSType:linux Architecture:x86_64 IndexServ
erAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:8 MemTotal:33662451712 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ubuntu-20-agent-10 Labels:[] ExperimentalBuild:false ServerVersion:20.10.17 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1 Expected:10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1} RuncCommit:{ID:v1.1.2-0-ga916309 Expected:v1.1.2-0-ga916309} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=default] ProductLicense: Warnings:<nil> ServerErrors:[] Clien
tInfo:{Debug:false Plugins:[map[Experimental:true Name:app Path:/usr/libexec/docker/cli-plugins/docker-app SchemaVersion:0.1.0 ShortDescription:Docker App Vendor:Docker Inc. Version:v0.9.1-beta3] map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.8.2-docker] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.6.0] map[Name:scan Path:/usr/libexec/docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.17.0]] Warnings:<nil>}}
I0630 19:35:10.782282 224099 start_flags.go:853] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0630 19:35:10.782321 224099 cni.go:95] Creating CNI manager for ""
I0630 19:35:10.782335 224099 cni.go:162] "docker" driver + containerd runtime found, recommending kindnet
I0630 19:35:10.782355 224099 start_flags.go:310] config:
{Name:default-k8s-different-port-20220630193354-10455 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.32-1656350719-14420@sha256:e7b7f38d1a2eba7828afc2c4c3d24e1d391db431976e47aa6dc5c7a6b038ca4e Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.24.2 ClusterName:default-k8s-different-port-20220630193354-10455 Namespace:default APIServerName:minikubeCA APIServerNames:
[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8444 NodeName:} Nodes:[{Name: IP:192.168.76.2 Port:8444 KubernetesVersion:v1.24.2 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[dashboard:true default-storageclass:true metrics-server:true storage-provisioner:true] CustomAddonImages:map[Dashboard:kubernetesui/dashboard:v2.6.0@sha256:4af9580485920635d888efe1eddbd67e12f9d5d84dba87100e93feb4e46636b3 MetricsScraper:k8s.gcr.io/echoserver:1.4 MetricsServer:k8s.gcr.io/echoserver:1.4] CustomAddonRegistries:map[MetricsServer:fake.domain] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAd
dress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath:}
I0630 19:35:10.787629 224099 out.go:177] * Starting control plane node default-k8s-different-port-20220630193354-10455 in cluster default-k8s-different-port-20220630193354-10455
I0630 19:35:10.789116 224099 cache.go:120] Beginning downloading kic base image for docker with containerd
I0630 19:35:10.790454 224099 out.go:177] * Pulling base image ...
I0630 19:35:10.791720 224099 preload.go:132] Checking if preload exists for k8s version v1.24.2 and runtime containerd
I0630 19:35:10.791761 224099 preload.go:148] Found local preload: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.24.2-containerd-overlay2-amd64.tar.lz4
I0630 19:35:10.791775 224099 cache.go:57] Caching tarball of preloaded images
I0630 19:35:10.791803 224099 image.go:75] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.32-1656350719-14420@sha256:e7b7f38d1a2eba7828afc2c4c3d24e1d391db431976e47aa6dc5c7a6b038ca4e in local docker daemon
I0630 19:35:10.792018 224099 preload.go:174] Found /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.24.2-containerd-overlay2-amd64.tar.lz4 in cache, skipping download
I0630 19:35:10.792036 224099 cache.go:60] Finished verifying existence of preloaded tar for v1.24.2 on containerd
I0630 19:35:10.792171 224099 profile.go:148] Saving config to /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/profiles/default-k8s-different-port-20220630193354-10455/config.json ...
I0630 19:35:10.829060 224099 image.go:79] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.32-1656350719-14420@sha256:e7b7f38d1a2eba7828afc2c4c3d24e1d391db431976e47aa6dc5c7a6b038ca4e in local docker daemon, skipping pull
I0630 19:35:10.829092 224099 cache.go:142] gcr.io/k8s-minikube/kicbase-builds:v0.0.32-1656350719-14420@sha256:e7b7f38d1a2eba7828afc2c4c3d24e1d391db431976e47aa6dc5c7a6b038ca4e exists in daemon, skipping load
I0630 19:35:10.829113 224099 cache.go:208] Successfully downloaded all kic artifacts
I0630 19:35:10.829159 224099 start.go:352] acquiring machines lock for default-k8s-different-port-20220630193354-10455: {Name:mk8dea4a26b6032885263b7012b1754d6271e0a7 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0630 19:35:10.829273 224099 start.go:356] acquired machines lock for "default-k8s-different-port-20220630193354-10455" in 84.54µs
I0630 19:35:10.829299 224099 start.go:94] Skipping create...Using existing machine configuration
I0630 19:35:10.829306 224099 fix.go:55] fixHost starting:
I0630 19:35:10.829604 224099 cli_runner.go:164] Run: docker container inspect default-k8s-different-port-20220630193354-10455 --format={{.State.Status}}
I0630 19:35:10.873348 224099 fix.go:103] recreateIfNeeded on default-k8s-different-port-20220630193354-10455: state=Stopped err=<nil>
W0630 19:35:10.873383 224099 fix.go:129] unexpected machine state, will restart: <nil>
I0630 19:35:10.877572 224099 out.go:177] * Restarting existing docker container for "default-k8s-different-port-20220630193354-10455" ...
I0630 19:35:09.465603 201441 pod_ready.go:102] pod "metrics-server-5c6f97fb75-4cdfg" in "kube-system" namespace has status "Ready":"False"
I0630 19:35:11.467145 201441 pod_ready.go:102] pod "metrics-server-5c6f97fb75-4cdfg" in "kube-system" namespace has status "Ready":"False"
I0630 19:35:09.532541 221899 api_server.go:165] Checking apiserver status ...
I0630 19:35:09.532614 221899 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
W0630 19:35:09.541030 221899 api_server.go:169] stopped: unable to get apiserver pid: sudo pgrep -xnf kube-apiserver.*minikube.*: Process exited with status 1
stdout:
stderr:
I0630 19:35:09.733320 221899 api_server.go:165] Checking apiserver status ...
I0630 19:35:09.733405 221899 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
W0630 19:35:09.741683 221899 api_server.go:169] stopped: unable to get apiserver pid: sudo pgrep -xnf kube-apiserver.*minikube.*: Process exited with status 1
stdout:
stderr:
I0630 19:35:09.741703 221899 api_server.go:165] Checking apiserver status ...
I0630 19:35:09.741740 221899 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
W0630 19:35:09.749318 221899 api_server.go:169] stopped: unable to get apiserver pid: sudo pgrep -xnf kube-apiserver.*minikube.*: Process exited with status 1
stdout:
stderr:
I0630 19:35:09.749342 221899 kubeadm.go:601] needs reconfigure: apiserver error: timed out waiting for the condition
I0630 19:35:09.749348 221899 kubeadm.go:1092] stopping kube-system containers ...
I0630 19:35:09.749360 221899 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name: Namespaces:[kube-system]}
I0630 19:35:09.749399 221899 ssh_runner.go:195] Run: sudo -s eval "crictl ps -a --quiet --label io.kubernetes.pod.namespace=kube-system"
I0630 19:35:09.772879 221899 cri.go:87] found id: "435510f20cf6a5730c5f463a7cb18adeee8e624aaf679a6aa52ba0308ff845e2"
I0630 19:35:09.772902 221899 cri.go:87] found id: "bd41da88adcd4ca8eec32645bbc45c40dd8cb3abd367c9f790ae312b20ffcbde"
I0630 19:35:09.772909 221899 cri.go:87] found id: "e586f98a235b3f4a1a1a63d05515776c3bef53817730934af28e4ab94ab23cf7"
I0630 19:35:09.772915 221899 cri.go:87] found id: "a657611603d0ff03028f91ef7b0e6145acac7bd0fbfc409a5b745b755225710f"
I0630 19:35:09.772920 221899 cri.go:87] found id: "77cf5004f2ac0c710a3cfd0a6940cde9eb69a3482608279b0c089758bd9ace6f"
I0630 19:35:09.772926 221899 cri.go:87] found id: "06db5c0d3d82317f2db60433b34e3f30894d8797c74f16fdf64b67f2e43583c1"
I0630 19:35:09.772932 221899 cri.go:87] found id: "210f453c0cff5b5b65abd4b0c27c06914d7103948e6de7231df89f44afece573"
I0630 19:35:09.772937 221899 cri.go:87] found id: "bbb3c400d8cd72e421aff239230301a9577c099cf0baa7827c30533df9836c3b"
I0630 19:35:09.772943 221899 cri.go:87] found id: ""
I0630 19:35:09.772947 221899 cri.go:232] Stopping containers: [435510f20cf6a5730c5f463a7cb18adeee8e624aaf679a6aa52ba0308ff845e2 bd41da88adcd4ca8eec32645bbc45c40dd8cb3abd367c9f790ae312b20ffcbde e586f98a235b3f4a1a1a63d05515776c3bef53817730934af28e4ab94ab23cf7 a657611603d0ff03028f91ef7b0e6145acac7bd0fbfc409a5b745b755225710f 77cf5004f2ac0c710a3cfd0a6940cde9eb69a3482608279b0c089758bd9ace6f 06db5c0d3d82317f2db60433b34e3f30894d8797c74f16fdf64b67f2e43583c1 210f453c0cff5b5b65abd4b0c27c06914d7103948e6de7231df89f44afece573 bbb3c400d8cd72e421aff239230301a9577c099cf0baa7827c30533df9836c3b]
I0630 19:35:09.772992 221899 ssh_runner.go:195] Run: which crictl
I0630 19:35:09.775793 221899 ssh_runner.go:195] Run: sudo /usr/bin/crictl stop 435510f20cf6a5730c5f463a7cb18adeee8e624aaf679a6aa52ba0308ff845e2 bd41da88adcd4ca8eec32645bbc45c40dd8cb3abd367c9f790ae312b20ffcbde e586f98a235b3f4a1a1a63d05515776c3bef53817730934af28e4ab94ab23cf7 a657611603d0ff03028f91ef7b0e6145acac7bd0fbfc409a5b745b755225710f 77cf5004f2ac0c710a3cfd0a6940cde9eb69a3482608279b0c089758bd9ace6f 06db5c0d3d82317f2db60433b34e3f30894d8797c74f16fdf64b67f2e43583c1 210f453c0cff5b5b65abd4b0c27c06914d7103948e6de7231df89f44afece573 bbb3c400d8cd72e421aff239230301a9577c099cf0baa7827c30533df9836c3b
I0630 19:35:09.800079 221899 ssh_runner.go:195] Run: sudo systemctl stop kubelet
I0630 19:35:09.809458 221899 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0630 19:35:09.815935 221899 kubeadm.go:155] found existing configuration files:
-rw------- 1 root root 5639 Jun 30 19:33 /etc/kubernetes/admin.conf
-rw------- 1 root root 5656 Jun 30 19:33 /etc/kubernetes/controller-manager.conf
-rw------- 1 root root 2067 Jun 30 19:33 /etc/kubernetes/kubelet.conf
-rw------- 1 root root 5604 Jun 30 19:33 /etc/kubernetes/scheduler.conf
I0630 19:35:09.815989 221899 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/admin.conf
I0630 19:35:09.822469 221899 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/kubelet.conf
I0630 19:35:09.828850 221899 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/controller-manager.conf
I0630 19:35:09.835145 221899 kubeadm.go:166] "https://control-plane.minikube.internal:8443" may not be in /etc/kubernetes/controller-manager.conf - will remove: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/controller-manager.conf: Process exited with status 1
stdout:
stderr:
I0630 19:35:09.835193 221899 ssh_runner.go:195] Run: sudo rm -f /etc/kubernetes/controller-manager.conf
I0630 19:35:09.841128 221899 ssh_runner.go:195] Run: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/scheduler.conf
I0630 19:35:09.847713 221899 kubeadm.go:166] "https://control-plane.minikube.internal:8443" may not be in /etc/kubernetes/scheduler.conf - will remove: sudo grep https://control-plane.minikube.internal:8443 /etc/kubernetes/scheduler.conf: Process exited with status 1
stdout:
stderr:
I0630 19:35:09.847765 221899 ssh_runner.go:195] Run: sudo rm -f /etc/kubernetes/scheduler.conf
I0630 19:35:09.854254 221899 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0630 19:35:09.861220 221899 kubeadm.go:703] reconfiguring cluster from /var/tmp/minikube/kubeadm.yaml
I0630 19:35:09.861243 221899 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init phase certs all --config /var/tmp/minikube/kubeadm.yaml"
I0630 19:35:09.915020 221899 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init phase kubeconfig all --config /var/tmp/minikube/kubeadm.yaml"
I0630 19:35:10.546527 221899 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init phase kubelet-start --config /var/tmp/minikube/kubeadm.yaml"
I0630 19:35:10.745071 221899 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init phase control-plane all --config /var/tmp/minikube/kubeadm.yaml"
I0630 19:35:10.803311 221899 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init phase etcd local --config /var/tmp/minikube/kubeadm.yaml"
I0630 19:35:10.935458 221899 api_server.go:51] waiting for apiserver process to appear ...
I0630 19:35:10.935522 221899 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:35:11.448114 221899 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:35:11.947574 221899 ssh_runner.go:195] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0630 19:35:12.030067 221899 api_server.go:71] duration metric: took 1.09461169s to wait for apiserver process to appear ...
I0630 19:35:12.030098 221899 api_server.go:87] waiting for apiserver healthz status ...
I0630 19:35:12.030110 221899 api_server.go:240] Checking apiserver healthz at https://192.168.85.2:8443/healthz ...
I0630 19:35:10.879539 224099 cli_runner.go:164] Run: docker start default-k8s-different-port-20220630193354-10455
I0630 19:35:11.288568 224099 cli_runner.go:164] Run: docker container inspect default-k8s-different-port-20220630193354-10455 --format={{.State.Status}}
I0630 19:35:11.325001 224099 kic.go:416] container "default-k8s-different-port-20220630193354-10455" state is running.
I0630 19:35:11.325538 224099 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" default-k8s-different-port-20220630193354-10455
I0630 19:35:11.360134 224099 profile.go:148] Saving config to /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/profiles/default-k8s-different-port-20220630193354-10455/config.json ...
I0630 19:35:11.360400 224099 machine.go:88] provisioning docker machine ...
I0630 19:35:11.360432 224099 ubuntu.go:169] provisioning hostname "default-k8s-different-port-20220630193354-10455"
I0630 19:35:11.360486 224099 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" default-k8s-different-port-20220630193354-10455
I0630 19:35:11.395948 224099 main.go:134] libmachine: Using SSH client type: native
I0630 19:35:11.396153 224099 main.go:134] libmachine: &{{{<nil> 0 [] [] []} docker [0x7dae00] 0x7dde60 <nil> [] 0s} 127.0.0.1 49402 <nil> <nil>}
I0630 19:35:11.396179 224099 main.go:134] libmachine: About to run SSH command:
sudo hostname default-k8s-different-port-20220630193354-10455 && echo "default-k8s-different-port-20220630193354-10455" | sudo tee /etc/hostname
I0630 19:35:11.396776 224099 main.go:134] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:36410->127.0.0.1:49402: read: connection reset by peer
I0630 19:35:14.522165 224099 main.go:134] libmachine: SSH cmd err, output: <nil>: default-k8s-different-port-20220630193354-10455
I0630 19:35:14.522236 224099 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" default-k8s-different-port-20220630193354-10455
I0630 19:35:14.556605 224099 main.go:134] libmachine: Using SSH client type: native
I0630 19:35:14.556816 224099 main.go:134] libmachine: &{{{<nil> 0 [] [] []} docker [0x7dae00] 0x7dde60 <nil> [] 0s} 127.0.0.1 49402 <nil> <nil>}
I0630 19:35:14.556844 224099 main.go:134] libmachine: About to run SSH command:
if ! grep -xq '.*\sdefault-k8s-different-port-20220630193354-10455' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 default-k8s-different-port-20220630193354-10455/g' /etc/hosts;
else
echo '127.0.1.1 default-k8s-different-port-20220630193354-10455' | sudo tee -a /etc/hosts;
fi
fi
I0630 19:35:14.681834 224099 main.go:134] libmachine: SSH cmd err, output: <nil>:
I0630 19:35:14.681872 224099 ubuntu.go:175] set auth options {CertDir:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube CaCertPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/ca.pem CaPrivateKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/server.pem ServerKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/server-key.pem ClientKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/c
erts/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube}
I0630 19:35:14.681898 224099 ubuntu.go:177] setting up certificates
I0630 19:35:14.681907 224099 provision.go:83] configureAuth start
I0630 19:35:14.681959 224099 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" default-k8s-different-port-20220630193354-10455
I0630 19:35:14.715308 224099 provision.go:138] copyHostCerts
I0630 19:35:14.715393 224099 exec_runner.go:144] found /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/ca.pem, removing ...
I0630 19:35:14.715414 224099 exec_runner.go:207] rm: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/ca.pem
I0630 19:35:14.715501 224099 exec_runner.go:151] cp: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/ca.pem --> /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/ca.pem (1082 bytes)
I0630 19:35:14.715609 224099 exec_runner.go:144] found /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cert.pem, removing ...
I0630 19:35:14.715629 224099 exec_runner.go:207] rm: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cert.pem
I0630 19:35:14.715671 224099 exec_runner.go:151] cp: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/cert.pem --> /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/cert.pem (1123 bytes)
I0630 19:35:14.715738 224099 exec_runner.go:144] found /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/key.pem, removing ...
I0630 19:35:14.715754 224099 exec_runner.go:207] rm: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/key.pem
I0630 19:35:14.715786 224099 exec_runner.go:151] cp: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/key.pem --> /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/key.pem (1679 bytes)
I0630 19:35:14.715849 224099 provision.go:112] generating server cert: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/server.pem ca-key=/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/ca.pem private-key=/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/ca-key.pem org=jenkins.default-k8s-different-port-20220630193354-10455 san=[192.168.76.2 127.0.0.1 localhost 127.0.0.1 minikube default-k8s-different-port-20220630193354-10455]
I0630 19:35:14.986551 224099 provision.go:172] copyRemoteCerts
I0630 19:35:14.986614 224099 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0630 19:35:14.986649 224099 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" default-k8s-different-port-20220630193354-10455
I0630 19:35:15.021360 224099 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49402 SSHKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/default-k8s-different-port-20220630193354-10455/id_rsa Username:docker}
I0630 19:35:15.110759 224099 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1082 bytes)
I0630 19:35:15.127108 224099 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/server.pem --> /etc/docker/server.pem (1310 bytes)
I0630 19:35:15.146168 224099 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1679 bytes)
I0630 19:35:15.164035 224099 provision.go:86] duration metric: configureAuth took 482.116893ms
I0630 19:35:15.164058 224099 ubuntu.go:193] setting minikube options for container-runtime
I0630 19:35:15.164225 224099 config.go:178] Loaded profile config "default-k8s-different-port-20220630193354-10455": Driver=docker, ContainerRuntime=containerd, KubernetesVersion=v1.24.2
I0630 19:35:15.164236 224099 machine.go:91] provisioned docker machine in 3.803817645s
I0630 19:35:15.164243 224099 start.go:306] post-start starting for "default-k8s-different-port-20220630193354-10455" (driver="docker")
I0630 19:35:15.164248 224099 start.go:316] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0630 19:35:15.164289 224099 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0630 19:35:15.164319 224099 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" default-k8s-different-port-20220630193354-10455
I0630 19:35:15.198205 224099 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49402 SSHKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/default-k8s-different-port-20220630193354-10455/id_rsa Username:docker}
I0630 19:35:15.281244 224099 ssh_runner.go:195] Run: cat /etc/os-release
I0630 19:35:15.283858 224099 main.go:134] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0630 19:35:15.283879 224099 main.go:134] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0630 19:35:15.283887 224099 main.go:134] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0630 19:35:15.283893 224099 info.go:137] Remote host: Ubuntu 20.04.4 LTS
I0630 19:35:15.283902 224099 filesync.go:126] Scanning /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/addons for local assets ...
I0630 19:35:15.283952 224099 filesync.go:126] Scanning /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/files for local assets ...
I0630 19:35:15.284042 224099 filesync.go:149] local asset: /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/files/etc/ssl/certs/104552.pem -> 104552.pem in /etc/ssl/certs
I0630 19:35:15.284126 224099 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0630 19:35:15.290754 224099 ssh_runner.go:362] scp /home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/files/etc/ssl/certs/104552.pem --> /etc/ssl/certs/104552.pem (1708 bytes)
I0630 19:35:15.307351 224099 start.go:309] post-start completed in 143.095263ms
I0630 19:35:15.307429 224099 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0630 19:35:15.307474 224099 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" default-k8s-different-port-20220630193354-10455
I0630 19:35:15.341732 224099 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49402 SSHKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/default-k8s-different-port-20220630193354-10455/id_rsa Username:docker}
I0630 19:35:15.422219 224099 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0630 19:35:15.426083 224099 fix.go:57] fixHost completed within 4.59677349s
I0630 19:35:15.426101 224099 start.go:81] releasing machines lock for "default-k8s-different-port-20220630193354-10455", held for 4.596812602s
I0630 19:35:15.426182 224099 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" default-k8s-different-port-20220630193354-10455
I0630 19:35:15.460674 224099 ssh_runner.go:195] Run: systemctl --version
I0630 19:35:15.460727 224099 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" default-k8s-different-port-20220630193354-10455
I0630 19:35:15.460758 224099 ssh_runner.go:195] Run: curl -sS -m 2 https://k8s.gcr.io/
I0630 19:35:15.460833 224099 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" default-k8s-different-port-20220630193354-10455
I0630 19:35:14.899905 221899 api_server.go:266] https://192.168.85.2:8443/healthz returned 403:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User \"system:anonymous\" cannot get path \"/healthz\"","reason":"Forbidden","details":{},"code":403}
W0630 19:35:14.899945 221899 api_server.go:102] status: https://192.168.85.2:8443/healthz returned error 403:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User \"system:anonymous\" cannot get path \"/healthz\"","reason":"Forbidden","details":{},"code":403}
I0630 19:35:15.400617 221899 api_server.go:240] Checking apiserver healthz at https://192.168.85.2:8443/healthz ...
I0630 19:35:15.405177 221899 api_server.go:266] https://192.168.85.2:8443/healthz returned 500:
[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/priority-and-fairness-config-consumer ok
[+]poststarthook/priority-and-fairness-filter ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[-]poststarthook/rbac/bootstrap-roles failed: reason withheld
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: reason withheld
[+]poststarthook/priority-and-fairness-config-producer ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/aggregator-reload-proxy-client-cert ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
[+]poststarthook/apiservice-openapiv3-controller ok
healthz check failed
W0630 19:35:15.405206 221899 api_server.go:102] status: https://192.168.85.2:8443/healthz returned error 500:
[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/priority-and-fairness-config-consumer ok
[+]poststarthook/priority-and-fairness-filter ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[-]poststarthook/rbac/bootstrap-roles failed: reason withheld
[-]poststarthook/scheduling/bootstrap-system-priority-classes failed: reason withheld
[+]poststarthook/priority-and-fairness-config-producer ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/aggregator-reload-proxy-client-cert ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
[+]poststarthook/apiservice-openapiv3-controller ok
healthz check failed
I0630 19:35:15.900775 221899 api_server.go:240] Checking apiserver healthz at https://192.168.85.2:8443/healthz ...
I0630 19:35:15.906992 221899 api_server.go:266] https://192.168.85.2:8443/healthz returned 500:
[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/priority-and-fairness-config-consumer ok
[+]poststarthook/priority-and-fairness-filter ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[-]poststarthook/rbac/bootstrap-roles failed: reason withheld
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/priority-and-fairness-config-producer ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/aggregator-reload-proxy-client-cert ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
[+]poststarthook/apiservice-openapiv3-controller ok
healthz check failed
W0630 19:35:15.907023 221899 api_server.go:102] status: https://192.168.85.2:8443/healthz returned error 500:
[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/priority-and-fairness-config-consumer ok
[+]poststarthook/priority-and-fairness-filter ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[-]poststarthook/rbac/bootstrap-roles failed: reason withheld
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/priority-and-fairness-config-producer ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/aggregator-reload-proxy-client-cert ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
[+]poststarthook/apiservice-openapiv3-controller ok
healthz check failed
I0630 19:35:16.400526 221899 api_server.go:240] Checking apiserver healthz at https://192.168.85.2:8443/healthz ...
I0630 19:35:16.405435 221899 api_server.go:266] https://192.168.85.2:8443/healthz returned 200:
ok
I0630 19:35:16.411129 221899 api_server.go:140] control plane version: v1.24.2
I0630 19:35:16.411152 221899 api_server.go:130] duration metric: took 4.381047234s to wait for apiserver health ...
I0630 19:35:16.411163 221899 cni.go:95] Creating CNI manager for ""
I0630 19:35:16.411172 221899 cni.go:162] "docker" driver + containerd runtime found, recommending kindnet
I0630 19:35:16.413215 221899 out.go:177] * Configuring CNI (Container Networking Interface) ...
I0630 19:35:13.966218 201441 pod_ready.go:102] pod "metrics-server-5c6f97fb75-4cdfg" in "kube-system" namespace has status "Ready":"False"
I0630 19:35:15.967182 201441 pod_ready.go:102] pod "metrics-server-5c6f97fb75-4cdfg" in "kube-system" namespace has status "Ready":"False"
I0630 19:35:16.414452 221899 ssh_runner.go:195] Run: stat /opt/cni/bin/portmap
I0630 19:35:16.417932 221899 cni.go:189] applying CNI manifest using /var/lib/minikube/binaries/v1.24.2/kubectl ...
I0630 19:35:16.417947 221899 ssh_runner.go:362] scp memory --> /var/tmp/minikube/cni.yaml (2429 bytes)
I0630 19:35:16.430647 221899 ssh_runner.go:195] Run: sudo /var/lib/minikube/binaries/v1.24.2/kubectl apply --kubeconfig=/var/lib/minikube/kubeconfig -f /var/tmp/minikube/cni.yaml
I0630 19:35:17.373535 221899 system_pods.go:43] waiting for kube-system pods to appear ...
I0630 19:35:17.435703 221899 system_pods.go:59] 9 kube-system pods found
I0630 19:35:17.435755 221899 system_pods.go:61] "coredns-6d4b75cb6d-zxsrx" [e7b20214-4a02-47f8-b15d-795beb91945d] Running
I0630 19:35:17.435770 221899 system_pods.go:61] "etcd-embed-certs-20220630193321-10455" [9daaec40-dc74-46a0-9bec-664b493a2df4] Running / Ready:ContainersNotReady (containers with unready status: [etcd]) / ContainersReady:ContainersNotReady (containers with unready status: [etcd])
I0630 19:35:17.435779 221899 system_pods.go:61] "kindnet-d78x4" [588b4229-bd6b-4e2b-83fe-ca3a01ef8374] Running
I0630 19:35:17.435787 221899 system_pods.go:61] "kube-apiserver-embed-certs-20220630193321-10455" [f68cdf06-2805-4fab-a166-8260f32faed6] Running
I0630 19:35:17.435799 221899 system_pods.go:61] "kube-controller-manager-embed-certs-20220630193321-10455" [a5eb5152-de87-4a58-84e9-f4a05c9f0a74] Running
I0630 19:35:17.435806 221899 system_pods.go:61] "kube-proxy-xgtdp" [10a104c3-cdef-4a0f-a8f0-ed49bddb7203] Running
I0630 19:35:17.435824 221899 system_pods.go:61] "kube-scheduler-embed-certs-20220630193321-10455" [39b6ef03-8a4c-4bd7-a182-1ed65423ca7a] Running / Ready:ContainersNotReady (containers with unready status: [kube-scheduler]) / ContainersReady:ContainersNotReady (containers with unready status: [kube-scheduler])
I0630 19:35:17.435835 221899 system_pods.go:61] "metrics-server-5c6f97fb75-wdlsp" [b0348ba6-f0cd-48d3-a481-937ef5c0318b] Pending / Ready:ContainersNotReady (containers with unready status: [metrics-server]) / ContainersReady:ContainersNotReady (containers with unready status: [metrics-server])
I0630 19:35:17.435848 221899 system_pods.go:61] "storage-provisioner" [a825d586-863b-4015-ae47-9416c4750728] Running
I0630 19:35:17.435856 221899 system_pods.go:74] duration metric: took 62.295045ms to wait for pod list to return data ...
I0630 19:35:17.435870 221899 node_conditions.go:102] verifying NodePressure condition ...
I0630 19:35:17.439587 221899 node_conditions.go:122] node storage ephemeral capacity is 304695084Ki
I0630 19:35:17.439636 221899 node_conditions.go:123] node cpu capacity is 8
I0630 19:35:17.439656 221899 node_conditions.go:105] duration metric: took 3.781441ms to run NodePressure ...
I0630 19:35:17.439676 221899 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init phase addon all --config /var/tmp/minikube/kubeadm.yaml"
I0630 19:35:17.861478 221899 kubeadm.go:762] waiting for restarted kubelet to initialise ...
I0630 19:35:17.866340 221899 kubeadm.go:777] kubelet initialised
I0630 19:35:17.866369 221899 kubeadm.go:778] duration metric: took 4.858086ms waiting for restarted kubelet to initialise ...
I0630 19:35:17.866380 221899 pod_ready.go:35] extra waiting up to 4m0s for all system-critical pods including labels [k8s-app=kube-dns component=etcd component=kube-apiserver component=kube-controller-manager k8s-app=kube-proxy component=kube-scheduler] to be "Ready" ...
I0630 19:35:17.933817 221899 pod_ready.go:78] waiting up to 4m0s for pod "coredns-6d4b75cb6d-zxsrx" in "kube-system" namespace to be "Ready" ...
I0630 19:35:15.496395 224099 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49402 SSHKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/default-k8s-different-port-20220630193354-10455/id_rsa Username:docker}
I0630 19:35:15.497605 224099 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49402 SSHKeyPath:/home/jenkins/minikube-integration/linux-amd64-docker-containerd-14420-3771-a1c71987949a080ebcb9d092f1fe4c348bd11abd/.minikube/machines/default-k8s-different-port-20220630193354-10455/id_rsa Username:docker}
I0630 19:35:15.577658 224099 ssh_runner.go:195] Run: sudo systemctl stop -f crio
I0630 19:35:15.599779 224099 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0630 19:35:15.609251 224099 docker.go:179] disabling docker service ...
I0630 19:35:15.609308 224099 ssh_runner.go:195] Run: sudo systemctl stop -f docker.socket
I0630 19:35:15.619466 224099 ssh_runner.go:195] Run: sudo systemctl stop -f docker.service
I0630 19:35:15.628892 224099 ssh_runner.go:195] Run: sudo systemctl disable docker.socket
I0630 19:35:15.706861 224099 ssh_runner.go:195] Run: sudo systemctl mask docker.service
I0630 19:35:15.784744 224099 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service docker
I0630 19:35:15.793648 224099 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %!s(MISSING) "runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0630 19:35:15.805605 224099 ssh_runner.go:195] Run: /bin/bash -c "sudo sed -e 's|^.*sandbox_image = .*$|sandbox_image = "k8s.gcr.io/pause:3.7"|' -i /etc/containerd/config.toml"
I0630 19:35:15.813279 224099 ssh_runner.go:195] Run: /bin/bash -c "sudo sed -e 's|^.*restrict_oom_score_adj = .*$|restrict_oom_score_adj = false|' -i /etc/containerd/config.toml"
I0630 19:35:15.820665 224099 ssh_runner.go:195] Run: /bin/bash -c "sudo sed -e 's|^.*SystemdCgroup = .*$|SystemdCgroup = false|' -i /etc/containerd/config.toml"
I0630 19:35:15.828392 224099 ssh_runner.go:195] Run: /bin/bash -c "sudo sed -e 's|^.*conf_dir = .*$|conf_dir = "/etc/cni/net.mk"|' -i /etc/containerd/config.toml"
I0630 19:35:15.835822 224099 ssh_runner.go:195] Run: /bin/bash -c "sudo sed -e 's|^# imports|imports = ["/etc/containerd/containerd.conf.d/02-containerd.conf"]|' -i /etc/containerd/config.toml"
I0630 19:35:15.843138 224099 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc/containerd/containerd.conf.d && printf %!s(MISSING) "dmVyc2lvbiA9IDIK" | base64 -d | sudo tee /etc/containerd/containerd.conf.d/02-containerd.conf"
I0630 19:35:15.855448 224099 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0630 19:35:15.863125 224099 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0630 19:35:15.869248 224099 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0630 19:35:15.953478 224099 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0630 19:35:16.038251 224099 start.go:447] Will wait 60s for socket path /run/containerd/containerd.sock
I0630 19:35:16.038326 224099 ssh_runner.go:195] Run: stat /run/containerd/containerd.sock
I0630 19:35:16.042594 224099 start.go:468] Will wait 60s for crictl version
I0630 19:35:16.042669 224099 ssh_runner.go:195] Run: sudo crictl version
I0630 19:35:16.074726 224099 retry.go:31] will retry after 11.04660288s: Temporary Error: sudo crictl version: Process exited with status 1
stdout:
stderr:
time="2022-06-30T19:35:16Z" level=fatal msg="getting the runtime version: rpc error: code = Unknown desc = server is not initialized yet"
I0630 19:35:20.321935 144400 kubeadm.go:397] StartCluster complete in 7m57.734528865s
I0630 19:35:20.321981 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-apiserver Namespaces:[]}
I0630 19:35:20.322027 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-apiserver
I0630 19:35:20.345506 144400 cri.go:87] found id: ""
I0630 19:35:20.345534 144400 logs.go:274] 0 containers: []
W0630 19:35:20.345542 144400 logs.go:276] No container was found matching "kube-apiserver"
I0630 19:35:20.345550 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:etcd Namespaces:[]}
I0630 19:35:20.345601 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=etcd
I0630 19:35:20.367499 144400 cri.go:87] found id: ""
I0630 19:35:20.367525 144400 logs.go:274] 0 containers: []
W0630 19:35:20.367531 144400 logs.go:276] No container was found matching "etcd"
I0630 19:35:20.367537 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:coredns Namespaces:[]}
I0630 19:35:20.367575 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=coredns
I0630 19:35:20.389868 144400 cri.go:87] found id: ""
I0630 19:35:20.389890 144400 logs.go:274] 0 containers: []
W0630 19:35:20.389896 144400 logs.go:276] No container was found matching "coredns"
I0630 19:35:20.389901 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-scheduler Namespaces:[]}
I0630 19:35:20.389940 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-scheduler
I0630 19:35:20.412692 144400 cri.go:87] found id: ""
I0630 19:35:20.412714 144400 logs.go:274] 0 containers: []
W0630 19:35:20.412719 144400 logs.go:276] No container was found matching "kube-scheduler"
I0630 19:35:20.412725 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-proxy Namespaces:[]}
I0630 19:35:20.412771 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-proxy
I0630 19:35:20.434617 144400 cri.go:87] found id: ""
I0630 19:35:20.434644 144400 logs.go:274] 0 containers: []
W0630 19:35:20.434652 144400 logs.go:276] No container was found matching "kube-proxy"
I0630 19:35:20.434659 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kubernetes-dashboard Namespaces:[]}
I0630 19:35:20.434701 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kubernetes-dashboard
I0630 19:35:20.459283 144400 cri.go:87] found id: ""
I0630 19:35:20.459304 144400 logs.go:274] 0 containers: []
W0630 19:35:20.459310 144400 logs.go:276] No container was found matching "kubernetes-dashboard"
I0630 19:35:20.459315 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:storage-provisioner Namespaces:[]}
I0630 19:35:20.459352 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=storage-provisioner
I0630 19:35:20.484451 144400 cri.go:87] found id: ""
I0630 19:35:20.484473 144400 logs.go:274] 0 containers: []
W0630 19:35:20.484479 144400 logs.go:276] No container was found matching "storage-provisioner"
I0630 19:35:20.484486 144400 cri.go:52] listing CRI containers in root /run/containerd/runc/k8s.io: {State:all Name:kube-controller-manager Namespaces:[]}
I0630 19:35:20.484535 144400 ssh_runner.go:195] Run: sudo crictl ps -a --quiet --name=kube-controller-manager
I0630 19:35:20.507241 144400 cri.go:87] found id: ""
I0630 19:35:20.507260 144400 logs.go:274] 0 containers: []
W0630 19:35:20.507268 144400 logs.go:276] No container was found matching "kube-controller-manager"
I0630 19:35:20.507277 144400 logs.go:123] Gathering logs for kubelet ...
I0630 19:35:20.507286 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0630 19:35:20.556138 144400 logs.go:138] Found kubelet problem: Jun 30 19:35:20 kubernetes-upgrade-20220630192547-10455 kubelet[11511]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:35:20.601617 144400 logs.go:123] Gathering logs for dmesg ...
I0630 19:35:20.601648 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0630 19:35:20.617961 144400 logs.go:123] Gathering logs for describe nodes ...
I0630 19:35:20.617992 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0630 19:35:20.665366 144400 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0630 19:35:20.665388 144400 logs.go:123] Gathering logs for containerd ...
I0630 19:35:20.665402 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u containerd -n 400"
I0630 19:35:20.708623 144400 logs.go:123] Gathering logs for container status ...
I0630 19:35:20.708655 144400 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
W0630 19:35:20.735172 144400 out.go:369] Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1012-gcp[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[0;37mCGROUPS_PIDS[0m: [0;32menabled[0m
[0;37mCGROUPS_HUGETLB[0m: [0;32menabled[0m
[0;37mCGROUPS_BLKIO[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'
stderr:
W0630 19:33:23.788059 9651 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1012-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0630 19:35:20.735211 144400 out.go:239] *
W0630 19:35:20.735427 144400 out.go:239] X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1012-gcp[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[0;37mCGROUPS_PIDS[0m: [0;32menabled[0m
[0;37mCGROUPS_HUGETLB[0m: [0;32menabled[0m
[0;37mCGROUPS_BLKIO[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'
stderr:
W0630 19:33:23.788059 9651 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1012-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0630 19:35:20.735465 144400 out.go:239] *
W0630 19:35:20.736207 144400 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0630 19:35:20.738341 144400 out.go:177] X Problems detected in kubelet:
I0630 19:35:20.740371 144400 out.go:177] Jun 30 19:35:20 kubernetes-upgrade-20220630192547-10455 kubelet[11511]: Error: failed to parse kubelet flag: unknown flag: --cni-conf-dir
I0630 19:35:20.743786 144400 out.go:177]
W0630 19:35:20.745382 144400 out.go:239] X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.24.2:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.24.2
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1012-gcp[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[0;37mCGROUPS_PIDS[0m: [0;32menabled[0m
[0;37mCGROUPS_HUGETLB[0m: [0;32menabled[0m
[0;37mCGROUPS_BLKIO[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI.
Here is one example how you may list all running Kubernetes containers by using crictl:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'
stderr:
W0630 19:33:23.788059 9651 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/containerd/containerd.sock". Please update your configuration!
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1012-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0630 19:35:20.745524 144400 out.go:239] * Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
W0630 19:35:20.745593 144400 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/4172
I0630 19:35:20.747207 144400 out.go:177]
*
* ==> container status <==
* CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
*
* ==> containerd <==
* -- Logs begin at Thu 2022-06-30 19:26:50 UTC, end at Thu 2022-06-30 19:35:21 UTC. --
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.485166807Z" level=error msg="StopPodSandbox for \"\\\"Using\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"\\\"Using\": not found"
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.503624407Z" level=info msg="StopPodSandbox for \"this\""
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.503691775Z" level=error msg="StopPodSandbox for \"this\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"this\": not found"
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.524899527Z" level=info msg="StopPodSandbox for \"endpoint\""
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.524963426Z" level=error msg="StopPodSandbox for \"endpoint\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"endpoint\": not found"
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.543850415Z" level=info msg="StopPodSandbox for \"is\""
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.543908012Z" level=error msg="StopPodSandbox for \"is\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"is\": not found"
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.564720214Z" level=info msg="StopPodSandbox for \"deprecated,\""
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.564777791Z" level=error msg="StopPodSandbox for \"deprecated,\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"deprecated,\": not found"
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.584780212Z" level=info msg="StopPodSandbox for \"please\""
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.584841724Z" level=error msg="StopPodSandbox for \"please\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"please\": not found"
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.605412441Z" level=info msg="StopPodSandbox for \"consider\""
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.605470872Z" level=error msg="StopPodSandbox for \"consider\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"consider\": not found"
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.625082446Z" level=info msg="StopPodSandbox for \"using\""
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.625152234Z" level=error msg="StopPodSandbox for \"using\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"using\": not found"
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.644882084Z" level=info msg="StopPodSandbox for \"full\""
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.644941294Z" level=error msg="StopPodSandbox for \"full\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"full\": not found"
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.662914053Z" level=info msg="StopPodSandbox for \"URL\""
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.662971636Z" level=error msg="StopPodSandbox for \"URL\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"URL\": not found"
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.683351627Z" level=info msg="StopPodSandbox for \"format\\\"\""
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.683416864Z" level=error msg="StopPodSandbox for \"format\\\"\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"format\\\"\": not found"
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.702670892Z" level=info msg="StopPodSandbox for \"endpoint=\\\"/run/containerd/containerd.sock\\\"\""
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.702728845Z" level=error msg="StopPodSandbox for \"endpoint=\\\"/run/containerd/containerd.sock\\\"\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"endpoint=\\\"/run/containerd/containerd.sock\\\"\": not found"
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.722078943Z" level=info msg="StopPodSandbox for \"URL=\\\"unix:///run/containerd/containerd.sock\\\"\""
Jun 30 19:33:23 kubernetes-upgrade-20220630192547-10455 containerd[494]: time="2022-06-30T19:33:23.722141559Z" level=error msg="StopPodSandbox for \"URL=\\\"unix:///run/containerd/containerd.sock\\\"\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find sandbox \"URL=\\\"unix:///run/containerd/containerd.sock\\\"\": not found"
*
* ==> describe nodes <==
*
* ==> dmesg <==
* [ +0.000002] ll header: 00000000: 02 42 05 63 92 37 02 42 c0 a8 4c 02 08 00
[ +0.000004] IPv4: martian source 10.96.0.1 from 10.244.0.2, on dev br-4d1f10f18ece
[ +0.000002] ll header: 00000000: 02 42 05 63 92 37 02 42 c0 a8 4c 02 08 00
[ +1.028738] IPv4: martian source 10.96.0.1 from 10.244.0.2, on dev br-4d1f10f18ece
[ +0.000001] IPv4: martian source 10.96.0.1 from 10.244.0.2, on dev br-4d1f10f18ece
[ +0.000005] ll header: 00000000: 02 42 05 63 92 37 02 42 c0 a8 4c 02 08 00
[ +0.000001] ll header: 00000000: 02 42 05 63 92 37 02 42 c0 a8 4c 02 08 00
[ +2.015859] IPv4: martian source 10.96.0.1 from 10.244.0.2, on dev br-4d1f10f18ece
[ +0.000009] ll header: 00000000: 02 42 05 63 92 37 02 42 c0 a8 4c 02 08 00
[ +0.004003] IPv4: martian source 10.96.0.1 from 10.244.0.2, on dev br-4d1f10f18ece
[ +0.000005] ll header: 00000000: 02 42 05 63 92 37 02 42 c0 a8 4c 02 08 00
[ +0.000297] IPv4: martian source 10.96.0.1 from 10.244.0.2, on dev br-4d1f10f18ece
[ +0.000004] ll header: 00000000: 02 42 05 63 92 37 02 42 c0 a8 4c 02 08 00
[ +4.155355] IPv4: martian source 10.96.0.1 from 10.244.0.2, on dev br-4d1f10f18ece
[ +0.000006] ll header: 00000000: 02 42 05 63 92 37 02 42 c0 a8 4c 02 08 00
[ +0.000011] IPv4: martian source 10.96.0.1 from 10.244.0.2, on dev br-4d1f10f18ece
[ +0.000005] ll header: 00000000: 02 42 05 63 92 37 02 42 c0 a8 4c 02 08 00
[ +0.000002] IPv4: martian source 10.96.0.1 from 10.244.0.2, on dev br-4d1f10f18ece
[ +0.000004] ll header: 00000000: 02 42 05 63 92 37 02 42 c0 a8 4c 02 08 00
[ +8.191363] IPv4: martian source 10.96.0.1 from 10.244.0.2, on dev br-4d1f10f18ece
[ +0.000008] ll header: 00000000: 02 42 05 63 92 37 02 42 c0 a8 4c 02 08 00
[ +0.000021] IPv4: martian source 10.96.0.1 from 10.244.0.2, on dev br-4d1f10f18ece
[ +0.000005] ll header: 00000000: 02 42 05 63 92 37 02 42 c0 a8 4c 02 08 00
[ +0.003972] IPv4: martian source 10.96.0.1 from 10.244.0.2, on dev br-4d1f10f18ece
[ +0.000005] ll header: 00000000: 02 42 05 63 92 37 02 42 c0 a8 4c 02 08 00
*
* ==> kernel <==
* 19:35:22 up 1:17, 0 users, load average: 2.06, 2.70, 2.00
Linux kubernetes-upgrade-20220630192547-10455 5.15.0-1012-gcp #17~20.04.1-Ubuntu SMP Thu Jun 23 16:10:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 20.04.4 LTS"
*
* ==> kubelet <==
* -- Logs begin at Thu 2022-06-30 19:26:50 UTC, end at Thu 2022-06-30 19:35:22 UTC. --
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --storage-driver-buffer-duration duration Writes in the storage driver will be buffered for this duration, and committed to the non memory backends as a single transaction (default 1m0s) (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --storage-driver-db string database name (default "cadvisor") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --storage-driver-host string database host:port (default "localhost:8086") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --storage-driver-password string database password (default "root") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --storage-driver-secure use secure connection with database (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --storage-driver-table string table name (default "stats") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --storage-driver-user string database username (default "root") (DEPRECATED: This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being removed.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --streaming-connection-idle-timeout duration Maximum time a streaming connection can be idle before the connection is automatically closed. 0 indicates no timeout. Example: '5m'. Note: All connections to the kubelet server have a maximum duration of 4 hours. (default 4h0m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --sync-frequency duration Max period between synchronizing running containers and config (default 1m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --system-cgroups string Optional absolute name of cgroups in which to place all non-kernel processes that are not already inside a cgroup under '/'. Empty for no container. Rolling back the flag requires a reboot. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --system-reserved mapStringString A set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=500Mi,ephemeral-storage=1Gi) pairs that describe resources reserved for non-kubernetes components. Currently only cpu and memory are supported. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for more detail. [default=none] (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --system-reserved-cgroup string Absolute name of the top level cgroup that is used to manage non-kubernetes components for which compute resources were reserved via '--system-reserved' flag. Ex. '/system-reserved'. [default=''] (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --tls-cert-file string File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert). If --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to --cert-dir. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --tls-cipher-suites strings Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384.
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --tls-min-version string Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13 (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --tls-private-key-file string File containing x509 private key matching --tls-cert-file. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --topology-manager-policy string Topology Manager policy to use. Possible values: 'none', 'best-effort', 'restricted', 'single-numa-node'. (default "none") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --topology-manager-scope string Scope to which topology hints applied. Topology Manager collects hints from Hint Providers and applies them to defined scope to ensure the pod admission. Possible values: 'container', 'pod'. (default "container") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: -v, --v Level number for the log level verbosity
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --version version[=true] Print version information and quit
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --vmodule pattern=N,... comma-separated list of pattern=N settings for file-filtered logging (only works for text log format)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --volume-plugin-dir string The full path of the directory in which to search for additional third party volume plugins (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
Jun 30 19:35:21 kubernetes-upgrade-20220630192547-10455 kubelet[11683]: --volume-stats-agg-period duration Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes. To disable volume calculations, set to a negative number. (default 1m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
-- /stdout --
** stderr **
E0630 19:35:21.993117 226655 logs.go:192] command /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" failed with error: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.24.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output: "\n** stderr ** \nThe connection to the server localhost:8443 was refused - did you specify the right host or port?\n\n** /stderr **"
! unable to fetch logs for: describe nodes
** /stderr **
helpers_test.go:254: (dbg) Run: out/minikube-linux-amd64 status --format={{.APIServer}} -p kubernetes-upgrade-20220630192547-10455 -n kubernetes-upgrade-20220630192547-10455
helpers_test.go:254: (dbg) Non-zero exit: out/minikube-linux-amd64 status --format={{.APIServer}} -p kubernetes-upgrade-20220630192547-10455 -n kubernetes-upgrade-20220630192547-10455: exit status 2 (390.614234ms)
-- stdout --
Stopped
-- /stdout --
helpers_test.go:254: status error: exit status 2 (may be ok)
helpers_test.go:256: "kubernetes-upgrade-20220630192547-10455" apiserver is not running, skipping kubectl commands (state="Stopped")
helpers_test.go:175: Cleaning up "kubernetes-upgrade-20220630192547-10455" profile ...
helpers_test.go:178: (dbg) Run: out/minikube-linux-amd64 delete -p kubernetes-upgrade-20220630192547-10455
helpers_test.go:178: (dbg) Done: out/minikube-linux-amd64 delete -p kubernetes-upgrade-20220630192547-10455: (2.19656324s)
--- FAIL: TestKubernetesUpgrade (576.68s)