=== RUN TestStartStop/group/old-k8s-version/serial/FirstStart
start_stop_delete_test.go:186: (dbg) Run: out/minikube-darwin-amd64 start -p old-k8s-version-221000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0
E0331 11:13:10.701653 2800 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/kindnet-346000/client.crt: no such file or directory
E0331 11:13:20.941313 2800 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/kindnet-346000/client.crt: no such file or directory
E0331 11:13:27.103639 2800 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/addons-841000/client.crt: no such file or directory
E0331 11:13:41.420889 2800 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/kindnet-346000/client.crt: no such file or directory
start_stop_delete_test.go:186: (dbg) Non-zero exit: out/minikube-darwin-amd64 start -p old-k8s-version-221000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0: exit status 109 (4m21.825121605s)
-- stdout --
* [old-k8s-version-221000] minikube v1.29.0 on Darwin 13.3
- MINIKUBE_LOCATION=16144
- KUBECONFIG=/Users/jenkins/minikube-integration/16144-2324/kubeconfig
- MINIKUBE_BIN=out/minikube-darwin-amd64
- MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
- MINIKUBE_HOME=/Users/jenkins/minikube-integration/16144-2324/.minikube
- MINIKUBE_FORCE_SYSTEMD=
* Using the docker driver based on user configuration
* Using Docker Desktop driver with root privileges
* Starting control plane node old-k8s-version-221000 in cluster old-k8s-version-221000
* Pulling base image ...
* Creating docker container (CPUs=2, Memory=2200MB) ...
* Preparing Kubernetes v1.16.0 on Docker 23.0.2 ...
- Generating certificates and keys ...
- Booting up control plane ...
- Generating certificates and keys ...
- Booting up control plane ...
-- /stdout --
** stderr **
I0331 11:13:06.366813 19978 out.go:296] Setting OutFile to fd 1 ...
I0331 11:13:06.367006 19978 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0331 11:13:06.367011 19978 out.go:309] Setting ErrFile to fd 2...
I0331 11:13:06.367015 19978 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0331 11:13:06.367131 19978 root.go:336] Updating PATH: /Users/jenkins/minikube-integration/16144-2324/.minikube/bin
I0331 11:13:06.368771 19978 out.go:303] Setting JSON to false
I0331 11:13:06.389972 19978 start.go:125] hostinfo: {"hostname":"MacOS-Agent-4.local","uptime":4354,"bootTime":1680282032,"procs":401,"os":"darwin","platform":"darwin","platformFamily":"Standalone Workstation","platformVersion":"13.3","kernelVersion":"22.4.0","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"f2f27e25-cfda-5ffd-9706-e98286194e62"}
W0331 11:13:06.390068 19978 start.go:133] gopshost.Virtualization returned error: not implemented yet
I0331 11:13:06.432495 19978 out.go:177] * [old-k8s-version-221000] minikube v1.29.0 on Darwin 13.3
I0331 11:13:06.453604 19978 out.go:177] - MINIKUBE_LOCATION=16144
I0331 11:13:06.453596 19978 notify.go:220] Checking for updates...
I0331 11:13:06.474439 19978 out.go:177] - KUBECONFIG=/Users/jenkins/minikube-integration/16144-2324/kubeconfig
I0331 11:13:06.516558 19978 out.go:177] - MINIKUBE_BIN=out/minikube-darwin-amd64
I0331 11:13:06.558442 19978 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0331 11:13:06.600300 19978 out.go:177] - MINIKUBE_HOME=/Users/jenkins/minikube-integration/16144-2324/.minikube
I0331 11:13:06.621515 19978 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0331 11:13:06.642875 19978 config.go:182] Loaded profile config "kubenet-346000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.26.3
I0331 11:13:06.642924 19978 driver.go:365] Setting default libvirt URI to qemu:///system
I0331 11:13:06.707865 19978 docker.go:121] docker version: linux-20.10.23:Docker Desktop 4.17.0 (99724)
I0331 11:13:06.707995 19978 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0331 11:13:06.900584 19978 info.go:266] docker info: {ID:7LJT:2NJA:NXZQ:FWAT:KIW7:M2WK:LGEH:GQAG:65D4:V5IZ:QKDO:7KKX Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:59 OomKillDisable:false NGoroutines:55 SystemTime:2023-03-31 18:13:06.760534754 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:5 KernelVersion:5.15.49-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServ
erAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:6 MemTotal:6231724032 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:20.10.23 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:2456e983eb9e37e47538f59ea18f2043c9a73640 Expected:2456e983eb9e37e47538f59ea18f2043c9a73640} RuncCommit:{ID:v1.1.4-0-g5fd4c4d Expected:v1.1.4-0-g5fd4c4d} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=defaul
t name=cgroupns] ProductLicense: Warnings:<nil> ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.10.3] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.15.1] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker Dev Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Manages Docker extensions Vendor:Docker Inc. Version:v0.2.18] map[Name:sbom Path
:/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM) for an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scan] ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.25.0] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Command line tool for Docker Scout Vendor:Docker Inc. Version:v0.6.0]] Warnings:<nil>}}
I0331 11:13:06.943009 19978 out.go:177] * Using the docker driver based on user configuration
I0331 11:13:06.963997 19978 start.go:295] selected driver: docker
I0331 11:13:06.964008 19978 start.go:859] validating driver "docker" against <nil>
I0331 11:13:06.964017 19978 start.go:870] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0331 11:13:06.967020 19978 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0331 11:13:07.167703 19978 info.go:266] docker info: {ID:7LJT:2NJA:NXZQ:FWAT:KIW7:M2WK:LGEH:GQAG:65D4:V5IZ:QKDO:7KKX Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:59 OomKillDisable:false NGoroutines:55 SystemTime:2023-03-31 18:13:07.02650783 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:5 KernelVersion:5.15.49-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServe
rAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:6 MemTotal:6231724032 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:20.10.23 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:2456e983eb9e37e47538f59ea18f2043c9a73640 Expected:2456e983eb9e37e47538f59ea18f2043c9a73640} RuncCommit:{ID:v1.1.4-0-g5fd4c4d Expected:v1.1.4-0-g5fd4c4d} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=default
name=cgroupns] ProductLicense: Warnings:<nil> ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.10.3] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.15.1] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker Dev Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Manages Docker extensions Vendor:Docker Inc. Version:v0.2.18] map[Name:sbom Path:
/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM) for an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scan] ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.25.0] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Command line tool for Docker Scout Vendor:Docker Inc. Version:v0.6.0]] Warnings:<nil>}}
I0331 11:13:07.167830 19978 start_flags.go:305] no existing cluster config was found, will generate one from the flags
I0331 11:13:07.168044 19978 start_flags.go:919] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0331 11:13:07.189455 19978 out.go:177] * Using Docker Desktop driver with root privileges
I0331 11:13:07.210222 19978 cni.go:84] Creating CNI manager for ""
I0331 11:13:07.210248 19978 cni.go:161] CNI unnecessary in this configuration, recommending no CNI
I0331 11:13:07.210254 19978 start_flags.go:319] config:
{Name:old-k8s-version-221000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.38@sha256:516db0892e1cd79b6781fc1a102fca4bf392576bbf3ca0fa01a467cb6cc0af55 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-221000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocke
t: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP:}
I0331 11:13:07.232271 19978 out.go:177] * Starting control plane node old-k8s-version-221000 in cluster old-k8s-version-221000
I0331 11:13:07.290245 19978 cache.go:120] Beginning downloading kic base image for docker with docker
I0331 11:13:07.311441 19978 out.go:177] * Pulling base image ...
I0331 11:13:07.370316 19978 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0331 11:13:07.370335 19978 image.go:79] Checking for gcr.io/k8s-minikube/kicbase:v0.0.38@sha256:516db0892e1cd79b6781fc1a102fca4bf392576bbf3ca0fa01a467cb6cc0af55 in local docker daemon
I0331 11:13:07.370430 19978 preload.go:148] Found local preload: /Users/jenkins/minikube-integration/16144-2324/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4
I0331 11:13:07.370453 19978 cache.go:57] Caching tarball of preloaded images
I0331 11:13:07.370707 19978 preload.go:174] Found /Users/jenkins/minikube-integration/16144-2324/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0331 11:13:07.370725 19978 cache.go:60] Finished verifying existence of preloaded tar for v1.16.0 on docker
I0331 11:13:07.371794 19978 profile.go:148] Saving config to /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/config.json ...
I0331 11:13:07.371947 19978 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/config.json: {Name:mkb5455ac9a0b555d1ca59ddce285d04990a845f Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0331 11:13:07.431063 19978 image.go:83] Found gcr.io/k8s-minikube/kicbase:v0.0.38@sha256:516db0892e1cd79b6781fc1a102fca4bf392576bbf3ca0fa01a467cb6cc0af55 in local docker daemon, skipping pull
I0331 11:13:07.431088 19978 cache.go:143] gcr.io/k8s-minikube/kicbase:v0.0.38@sha256:516db0892e1cd79b6781fc1a102fca4bf392576bbf3ca0fa01a467cb6cc0af55 exists in daemon, skipping load
I0331 11:13:07.431110 19978 cache.go:193] Successfully downloaded all kic artifacts
I0331 11:13:07.431167 19978 start.go:364] acquiring machines lock for old-k8s-version-221000: {Name:mkd3c9d5738895d94e9fe50102426daf0ea0e9c5 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0331 11:13:07.431328 19978 start.go:368] acquired machines lock for "old-k8s-version-221000" in 148.215µs
I0331 11:13:07.431357 19978 start.go:93] Provisioning new machine with config: &{Name:old-k8s-version-221000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.38@sha256:516db0892e1cd79b6781fc1a102fca4bf392576bbf3ca0fa01a467cb6cc0af55 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-221000 Namespace:default APIServerName:minikubeCA APIServe
rNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:fals
e CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP:} &{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}
I0331 11:13:07.431417 19978 start.go:125] createHost starting for "" (driver="docker")
I0331 11:13:07.453178 19978 out.go:204] * Creating docker container (CPUs=2, Memory=2200MB) ...
I0331 11:13:07.453409 19978 start.go:159] libmachine.API.Create for "old-k8s-version-221000" (driver="docker")
I0331 11:13:07.453426 19978 client.go:168] LocalClient.Create starting
I0331 11:13:07.453537 19978 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/16144-2324/.minikube/certs/ca.pem
I0331 11:13:07.453580 19978 main.go:141] libmachine: Decoding PEM data...
I0331 11:13:07.453598 19978 main.go:141] libmachine: Parsing certificate...
I0331 11:13:07.453662 19978 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/16144-2324/.minikube/certs/cert.pem
I0331 11:13:07.453686 19978 main.go:141] libmachine: Decoding PEM data...
I0331 11:13:07.453702 19978 main.go:141] libmachine: Parsing certificate...
I0331 11:13:07.454169 19978 cli_runner.go:164] Run: docker network inspect old-k8s-version-221000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
W0331 11:13:07.512466 19978 cli_runner.go:211] docker network inspect old-k8s-version-221000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1
I0331 11:13:07.512573 19978 network_create.go:281] running [docker network inspect old-k8s-version-221000] to gather additional debugging logs...
I0331 11:13:07.512596 19978 cli_runner.go:164] Run: docker network inspect old-k8s-version-221000
W0331 11:13:07.571232 19978 cli_runner.go:211] docker network inspect old-k8s-version-221000 returned with exit code 1
I0331 11:13:07.571262 19978 network_create.go:284] error running [docker network inspect old-k8s-version-221000]: docker network inspect old-k8s-version-221000: exit status 1
stdout:
[]
stderr:
Error: No such network: old-k8s-version-221000
I0331 11:13:07.571277 19978 network_create.go:286] output of [docker network inspect old-k8s-version-221000]: -- stdout --
[]
-- /stdout --
** stderr **
Error: No such network: old-k8s-version-221000
** /stderr **
I0331 11:13:07.571364 19978 cli_runner.go:164] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0331 11:13:07.631434 19978 network.go:212] skipping subnet 192.168.49.0/24 that is reserved: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0331 11:13:07.632850 19978 network.go:212] skipping subnet 192.168.58.0/24 that is reserved: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0331 11:13:07.634184 19978 network.go:212] skipping subnet 192.168.67.0/24 that is reserved: &{IP:192.168.67.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.67.0/24 Gateway:192.168.67.1 ClientMin:192.168.67.2 ClientMax:192.168.67.254 Broadcast:192.168.67.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0331 11:13:07.634536 19978 network.go:209] using free private subnet 192.168.76.0/24: &{IP:192.168.76.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.76.0/24 Gateway:192.168.76.1 ClientMin:192.168.76.2 ClientMax:192.168.76.254 Broadcast:192.168.76.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc000e19200}
I0331 11:13:07.634551 19978 network_create.go:123] attempt to create docker network old-k8s-version-221000 192.168.76.0/24 with gateway 192.168.76.1 and MTU of 1500 ...
I0331 11:13:07.634632 19978 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.76.0/24 --gateway=192.168.76.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=1500 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-221000 old-k8s-version-221000
I0331 11:13:07.727702 19978 network_create.go:107] docker network old-k8s-version-221000 192.168.76.0/24 created
I0331 11:13:07.727740 19978 kic.go:117] calculated static IP "192.168.76.2" for the "old-k8s-version-221000" container
I0331 11:13:07.727867 19978 cli_runner.go:164] Run: docker ps -a --format {{.Names}}
I0331 11:13:07.787942 19978 cli_runner.go:164] Run: docker volume create old-k8s-version-221000 --label name.minikube.sigs.k8s.io=old-k8s-version-221000 --label created_by.minikube.sigs.k8s.io=true
I0331 11:13:07.847367 19978 oci.go:103] Successfully created a docker volume old-k8s-version-221000
I0331 11:13:07.847501 19978 cli_runner.go:164] Run: docker run --rm --name old-k8s-version-221000-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-221000 --entrypoint /usr/bin/test -v old-k8s-version-221000:/var gcr.io/k8s-minikube/kicbase:v0.0.38@sha256:516db0892e1cd79b6781fc1a102fca4bf392576bbf3ca0fa01a467cb6cc0af55 -d /var/lib
I0331 11:13:08.414346 19978 oci.go:107] Successfully prepared a docker volume old-k8s-version-221000
I0331 11:13:08.414379 19978 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0331 11:13:08.414393 19978 kic.go:190] Starting extracting preloaded images to volume ...
I0331 11:13:08.414502 19978 cli_runner.go:164] Run: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/16144-2324/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-221000:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.38@sha256:516db0892e1cd79b6781fc1a102fca4bf392576bbf3ca0fa01a467cb6cc0af55 -I lz4 -xf /preloaded.tar -C /extractDir
I0331 11:13:14.536430 19978 cli_runner.go:217] Completed: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/16144-2324/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-221000:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.38@sha256:516db0892e1cd79b6781fc1a102fca4bf392576bbf3ca0fa01a467cb6cc0af55 -I lz4 -xf /preloaded.tar -C /extractDir: (6.122151883s)
I0331 11:13:14.536462 19978 kic.go:199] duration metric: took 6.122373 seconds to extract preloaded images to volume
I0331 11:13:14.536585 19978 cli_runner.go:164] Run: docker info --format "'{{json .SecurityOptions}}'"
I0331 11:13:14.729097 19978 cli_runner.go:164] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname old-k8s-version-221000 --name old-k8s-version-221000 --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-221000 --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=old-k8s-version-221000 --network old-k8s-version-221000 --ip 192.168.76.2 --volume old-k8s-version-221000:/var --security-opt apparmor=unconfined --memory=2200mb --memory-swap=2200mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.38@sha256:516db0892e1cd79b6781fc1a102fca4bf392576bbf3ca0fa01a467cb6cc0af55
I0331 11:13:15.112821 19978 cli_runner.go:164] Run: docker container inspect old-k8s-version-221000 --format={{.State.Running}}
I0331 11:13:15.175801 19978 cli_runner.go:164] Run: docker container inspect old-k8s-version-221000 --format={{.State.Status}}
I0331 11:13:15.245219 19978 cli_runner.go:164] Run: docker exec old-k8s-version-221000 stat /var/lib/dpkg/alternatives/iptables
I0331 11:13:15.373092 19978 oci.go:144] the created container "old-k8s-version-221000" has a running status.
I0331 11:13:15.373129 19978 kic.go:221] Creating ssh key for kic: /Users/jenkins/minikube-integration/16144-2324/.minikube/machines/old-k8s-version-221000/id_rsa...
I0331 11:13:15.498250 19978 kic_runner.go:191] docker (temp): /Users/jenkins/minikube-integration/16144-2324/.minikube/machines/old-k8s-version-221000/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0331 11:13:15.617525 19978 cli_runner.go:164] Run: docker container inspect old-k8s-version-221000 --format={{.State.Status}}
I0331 11:13:15.682168 19978 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0331 11:13:15.682189 19978 kic_runner.go:114] Args: [docker exec --privileged old-k8s-version-221000 chown docker:docker /home/docker/.ssh/authorized_keys]
I0331 11:13:15.800490 19978 cli_runner.go:164] Run: docker container inspect old-k8s-version-221000 --format={{.State.Status}}
I0331 11:13:15.867192 19978 machine.go:88] provisioning docker machine ...
I0331 11:13:15.867235 19978 ubuntu.go:169] provisioning hostname "old-k8s-version-221000"
I0331 11:13:15.867336 19978 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-221000
I0331 11:13:15.931308 19978 main.go:141] libmachine: Using SSH client type: native
I0331 11:13:15.931720 19978 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x140cc00] 0x140fca0 <nil> [] 0s} 127.0.0.1 53352 <nil> <nil>}
I0331 11:13:15.931735 19978 main.go:141] libmachine: About to run SSH command:
sudo hostname old-k8s-version-221000 && echo "old-k8s-version-221000" | sudo tee /etc/hostname
I0331 11:13:16.076684 19978 main.go:141] libmachine: SSH cmd err, output: <nil>: old-k8s-version-221000
I0331 11:13:16.076785 19978 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-221000
I0331 11:13:16.137221 19978 main.go:141] libmachine: Using SSH client type: native
I0331 11:13:16.137564 19978 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x140cc00] 0x140fca0 <nil> [] 0s} 127.0.0.1 53352 <nil> <nil>}
I0331 11:13:16.137582 19978 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sold-k8s-version-221000' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 old-k8s-version-221000/g' /etc/hosts;
else
echo '127.0.1.1 old-k8s-version-221000' | sudo tee -a /etc/hosts;
fi
fi
I0331 11:13:16.269208 19978 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0331 11:13:16.269233 19978 ubuntu.go:175] set auth options {CertDir:/Users/jenkins/minikube-integration/16144-2324/.minikube CaCertPath:/Users/jenkins/minikube-integration/16144-2324/.minikube/certs/ca.pem CaPrivateKeyPath:/Users/jenkins/minikube-integration/16144-2324/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/Users/jenkins/minikube-integration/16144-2324/.minikube/machines/server.pem ServerKeyPath:/Users/jenkins/minikube-integration/16144-2324/.minikube/machines/server-key.pem ClientKeyPath:/Users/jenkins/minikube-integration/16144-2324/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/Users/jenkins/minikube-integration/16144-2324/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/Users/jenkins/minikube-integration/16144-2324/.minikube}
I0331 11:13:16.269263 19978 ubuntu.go:177] setting up certificates
I0331 11:13:16.269274 19978 provision.go:83] configureAuth start
I0331 11:13:16.269365 19978 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-221000
I0331 11:13:16.332988 19978 provision.go:138] copyHostCerts
I0331 11:13:16.333090 19978 exec_runner.go:144] found /Users/jenkins/minikube-integration/16144-2324/.minikube/ca.pem, removing ...
I0331 11:13:16.333098 19978 exec_runner.go:207] rm: /Users/jenkins/minikube-integration/16144-2324/.minikube/ca.pem
I0331 11:13:16.333203 19978 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/16144-2324/.minikube/certs/ca.pem --> /Users/jenkins/minikube-integration/16144-2324/.minikube/ca.pem (1078 bytes)
I0331 11:13:16.333402 19978 exec_runner.go:144] found /Users/jenkins/minikube-integration/16144-2324/.minikube/cert.pem, removing ...
I0331 11:13:16.333408 19978 exec_runner.go:207] rm: /Users/jenkins/minikube-integration/16144-2324/.minikube/cert.pem
I0331 11:13:16.333468 19978 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/16144-2324/.minikube/certs/cert.pem --> /Users/jenkins/minikube-integration/16144-2324/.minikube/cert.pem (1123 bytes)
I0331 11:13:16.333650 19978 exec_runner.go:144] found /Users/jenkins/minikube-integration/16144-2324/.minikube/key.pem, removing ...
I0331 11:13:16.333656 19978 exec_runner.go:207] rm: /Users/jenkins/minikube-integration/16144-2324/.minikube/key.pem
I0331 11:13:16.333714 19978 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/16144-2324/.minikube/certs/key.pem --> /Users/jenkins/minikube-integration/16144-2324/.minikube/key.pem (1679 bytes)
I0331 11:13:16.333840 19978 provision.go:112] generating server cert: /Users/jenkins/minikube-integration/16144-2324/.minikube/machines/server.pem ca-key=/Users/jenkins/minikube-integration/16144-2324/.minikube/certs/ca.pem private-key=/Users/jenkins/minikube-integration/16144-2324/.minikube/certs/ca-key.pem org=jenkins.old-k8s-version-221000 san=[192.168.76.2 127.0.0.1 localhost 127.0.0.1 minikube old-k8s-version-221000]
I0331 11:13:16.463686 19978 provision.go:172] copyRemoteCerts
I0331 11:13:16.463748 19978 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0331 11:13:16.463796 19978 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-221000
I0331 11:13:16.525258 19978 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:53352 SSHKeyPath:/Users/jenkins/minikube-integration/16144-2324/.minikube/machines/old-k8s-version-221000/id_rsa Username:docker}
I0331 11:13:16.619636 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1078 bytes)
I0331 11:13:16.637372 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/machines/server.pem --> /etc/docker/server.pem (1241 bytes)
I0331 11:13:16.655080 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1679 bytes)
I0331 11:13:16.672941 19978 provision.go:86] duration metric: configureAuth took 403.671759ms
I0331 11:13:16.672957 19978 ubuntu.go:193] setting minikube options for container-runtime
I0331 11:13:16.673117 19978 config.go:182] Loaded profile config "old-k8s-version-221000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.16.0
I0331 11:13:16.673178 19978 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-221000
I0331 11:13:16.733974 19978 main.go:141] libmachine: Using SSH client type: native
I0331 11:13:16.734320 19978 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x140cc00] 0x140fca0 <nil> [] 0s} 127.0.0.1 53352 <nil> <nil>}
I0331 11:13:16.734333 19978 main.go:141] libmachine: About to run SSH command:
df --output=fstype / | tail -n 1
I0331 11:13:16.868597 19978 main.go:141] libmachine: SSH cmd err, output: <nil>: overlay
I0331 11:13:16.868612 19978 ubuntu.go:71] root file system type: overlay
I0331 11:13:16.868703 19978 provision.go:309] Updating docker unit: /lib/systemd/system/docker.service ...
I0331 11:13:16.868809 19978 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-221000
I0331 11:13:16.929717 19978 main.go:141] libmachine: Using SSH client type: native
I0331 11:13:16.930072 19978 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x140cc00] 0x140fca0 <nil> [] 0s} 127.0.0.1 53352 <nil> <nil>}
I0331 11:13:16.930119 19978 main.go:141] libmachine: About to run SSH command:
sudo mkdir -p /lib/systemd/system && printf %s "[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP \$MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service.new
I0331 11:13:17.074987 19978 main.go:141] libmachine: SSH cmd err, output: <nil>: [Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
I0331 11:13:17.075083 19978 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-221000
I0331 11:13:17.135312 19978 main.go:141] libmachine: Using SSH client type: native
I0331 11:13:17.135649 19978 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x140cc00] 0x140fca0 <nil> [] 0s} 127.0.0.1 53352 <nil> <nil>}
I0331 11:13:17.135661 19978 main.go:141] libmachine: About to run SSH command:
sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
I0331 11:13:17.797400 19978 main.go:141] libmachine: SSH cmd err, output: <nil>: --- /lib/systemd/system/docker.service 2023-03-27 16:16:18.000000000 +0000
+++ /lib/systemd/system/docker.service.new 2023-03-31 18:13:17.073278828 +0000
@@ -1,30 +1,32 @@
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
-After=network-online.target docker.socket firewalld.service containerd.service time-set.target
-Wants=network-online.target containerd.service
+BindsTo=containerd.service
+After=network-online.target firewalld.service containerd.service
+Wants=network-online.target
Requires=docker.socket
+StartLimitBurst=3
+StartLimitIntervalSec=60
[Service]
Type=notify
-# the default is not to use systemd for cgroups because the delegate issues still
-# exists and systemd currently does not support the cgroup feature set required
-# for containers run by docker
-ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
-ExecReload=/bin/kill -s HUP $MAINPID
-TimeoutStartSec=0
-RestartSec=2
-Restart=always
+Restart=on-failure
-# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
-# Both the old, and new location are accepted by systemd 229 and up, so using the old location
-# to make them work for either version of systemd.
-StartLimitBurst=3
-# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
-# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
-# this option work for either version of systemd.
-StartLimitInterval=60s
+
+# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
+# The base configuration already specifies an 'ExecStart=...' command. The first directive
+# here is to clear out that command inherited from the base configuration. Without this,
+# the command from the base configuration and the command specified here are treated as
+# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
+# will catch this invalid input and refuse to start the service with an error like:
+# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
+
+# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
+# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
+ExecStart=
+ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
+ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
@@ -32,16 +34,16 @@
LimitNPROC=infinity
LimitCORE=infinity
-# Comment TasksMax if your systemd version does not support it.
-# Only systemd 226 and above support this option.
+# Uncomment TasksMax if your systemd version supports it.
+# Only systemd 226 and above support this version.
TasksMax=infinity
+TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
-OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
I0331 11:13:17.797429 19978 machine.go:91] provisioned docker machine in 1.93031044s
I0331 11:13:17.797441 19978 client.go:171] LocalClient.Create took 10.344524886s
I0331 11:13:17.797461 19978 start.go:167] duration metric: libmachine.API.Create for "old-k8s-version-221000" took 10.34456762s
I0331 11:13:17.797523 19978 start.go:300] post-start starting for "old-k8s-version-221000" (driver="docker")
I0331 11:13:17.797537 19978 start.go:328] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0331 11:13:17.797657 19978 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0331 11:13:17.797747 19978 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-221000
I0331 11:13:17.862681 19978 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:53352 SSHKeyPath:/Users/jenkins/minikube-integration/16144-2324/.minikube/machines/old-k8s-version-221000/id_rsa Username:docker}
I0331 11:13:17.958975 19978 ssh_runner.go:195] Run: cat /etc/os-release
I0331 11:13:17.962802 19978 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0331 11:13:17.962817 19978 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0331 11:13:17.962825 19978 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0331 11:13:17.962831 19978 info.go:137] Remote host: Ubuntu 20.04.5 LTS
I0331 11:13:17.962843 19978 filesync.go:126] Scanning /Users/jenkins/minikube-integration/16144-2324/.minikube/addons for local assets ...
I0331 11:13:17.962929 19978 filesync.go:126] Scanning /Users/jenkins/minikube-integration/16144-2324/.minikube/files for local assets ...
I0331 11:13:17.963109 19978 filesync.go:149] local asset: /Users/jenkins/minikube-integration/16144-2324/.minikube/files/etc/ssl/certs/28002.pem -> 28002.pem in /etc/ssl/certs
I0331 11:13:17.963317 19978 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0331 11:13:17.971195 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/files/etc/ssl/certs/28002.pem --> /etc/ssl/certs/28002.pem (1708 bytes)
I0331 11:13:17.989376 19978 start.go:303] post-start completed in 191.847185ms
I0331 11:13:17.997975 19978 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-221000
I0331 11:13:18.060143 19978 profile.go:148] Saving config to /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/config.json ...
I0331 11:13:18.060686 19978 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0331 11:13:18.060745 19978 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-221000
I0331 11:13:18.121624 19978 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:53352 SSHKeyPath:/Users/jenkins/minikube-integration/16144-2324/.minikube/machines/old-k8s-version-221000/id_rsa Username:docker}
I0331 11:13:18.212878 19978 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0331 11:13:18.217690 19978 start.go:128] duration metric: createHost completed in 10.786793684s
I0331 11:13:18.217712 19978 start.go:83] releasing machines lock for "old-k8s-version-221000", held for 10.786916615s
I0331 11:13:18.217813 19978 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-221000
I0331 11:13:18.277678 19978 ssh_runner.go:195] Run: cat /version.json
I0331 11:13:18.277699 19978 ssh_runner.go:195] Run: curl -sS -m 2 https://registry.k8s.io/
I0331 11:13:18.277749 19978 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-221000
I0331 11:13:18.277781 19978 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-221000
I0331 11:13:18.341358 19978 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:53352 SSHKeyPath:/Users/jenkins/minikube-integration/16144-2324/.minikube/machines/old-k8s-version-221000/id_rsa Username:docker}
I0331 11:13:18.341824 19978 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:53352 SSHKeyPath:/Users/jenkins/minikube-integration/16144-2324/.minikube/machines/old-k8s-version-221000/id_rsa Username:docker}
W0331 11:13:18.488961 19978 out.go:239] ! Image was not built for the current minikube version. To resolve this you can delete and recreate your minikube cluster using the latest images. Expected minikube version: v1.30.0 -> Actual minikube version: v1.29.0
! Image was not built for the current minikube version. To resolve this you can delete and recreate your minikube cluster using the latest images. Expected minikube version: v1.30.0 -> Actual minikube version: v1.29.0
I0331 11:13:18.489043 19978 ssh_runner.go:195] Run: systemctl --version
I0331 11:13:18.494470 19978 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
I0331 11:13:18.499762 19978 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *loopback.conf* -not -name *.mk_disabled -exec sh -c "grep -q loopback {} && ( grep -q name {} || sudo sed -i '/"type": "loopback"/i \ \ \ \ "name": "loopback",' {} ) && sudo sed -i 's|"cniVersion": ".*"|"cniVersion": "1.0.0"|g' {}" ;
I0331 11:13:18.520765 19978 cni.go:229] loopback cni configuration patched: "/etc/cni/net.d/*loopback.conf*" found
I0331 11:13:18.520841 19978 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *bridge* -not -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e '/"dst": ".*:.*"/d' -e 's|^(.*)"dst": (.*)[,*]$|\1"dst": \2|g' -e '/"subnet": ".*:.*"/d' -e 's|^(.*)"subnet": ".*"(.*)[,*]$|\1"subnet": "10.244.0.0/16"\2|g' {}" ;
I0331 11:13:18.534941 19978 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e 's|^(.*)"subnet": ".*"(.*)$|\1"subnet": "10.244.0.0/16"\2|g' -e 's|^(.*)"gateway": ".*"(.*)$|\1"gateway": "10.244.0.1"\2|g' {}" ;
I0331 11:13:18.543008 19978 cni.go:307] configured [/etc/cni/net.d/100-crio-bridge.conf] bridge cni config(s)
I0331 11:13:18.543025 19978 start.go:481] detecting cgroup driver to use...
I0331 11:13:18.543041 19978 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0331 11:13:18.543109 19978 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0331 11:13:18.556581 19978 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.1"|' /etc/containerd/config.toml"
I0331 11:13:18.565531 19978 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I0331 11:13:18.574598 19978 containerd.go:145] configuring containerd to use "cgroupfs" as cgroup driver...
I0331 11:13:18.574660 19978 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = false|g' /etc/containerd/config.toml"
I0331 11:13:18.583762 19978 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0331 11:13:18.593242 19978 ssh_runner.go:195] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I0331 11:13:18.603105 19978 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0331 11:13:18.612723 19978 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0331 11:13:18.621917 19978 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I0331 11:13:18.632133 19978 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0331 11:13:18.640431 19978 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0331 11:13:18.648267 19978 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0331 11:13:18.720459 19978 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0331 11:13:18.795434 19978 start.go:481] detecting cgroup driver to use...
I0331 11:13:18.795456 19978 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0331 11:13:18.795524 19978 ssh_runner.go:195] Run: sudo systemctl cat docker.service
I0331 11:13:18.806055 19978 cruntime.go:276] skipping containerd shutdown because we are bound to it
I0331 11:13:18.806127 19978 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0331 11:13:18.818521 19978 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///var/run/dockershim.sock
" | sudo tee /etc/crictl.yaml"
I0331 11:13:18.833004 19978 ssh_runner.go:195] Run: which cri-dockerd
I0331 11:13:18.837503 19978 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/cri-docker.service.d
I0331 11:13:18.846089 19978 ssh_runner.go:362] scp memory --> /etc/systemd/system/cri-docker.service.d/10-cni.conf (189 bytes)
I0331 11:13:18.860963 19978 ssh_runner.go:195] Run: sudo systemctl unmask docker.service
I0331 11:13:18.937504 19978 ssh_runner.go:195] Run: sudo systemctl enable docker.socket
I0331 11:13:19.027337 19978 docker.go:538] configuring docker to use "cgroupfs" as cgroup driver...
I0331 11:13:19.027352 19978 ssh_runner.go:362] scp memory --> /etc/docker/daemon.json (144 bytes)
I0331 11:13:19.041404 19978 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0331 11:13:19.135152 19978 ssh_runner.go:195] Run: sudo systemctl restart docker
I0331 11:13:19.379720 19978 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0331 11:13:19.406874 19978 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0331 11:13:19.454909 19978 out.go:204] * Preparing Kubernetes v1.16.0 on Docker 23.0.2 ...
I0331 11:13:19.455066 19978 cli_runner.go:164] Run: docker exec -t old-k8s-version-221000 dig +short host.docker.internal
I0331 11:13:19.576560 19978 network.go:96] got host ip for mount in container by digging dns: 192.168.65.2
I0331 11:13:19.576678 19978 ssh_runner.go:195] Run: grep 192.168.65.2 host.minikube.internal$ /etc/hosts
I0331 11:13:19.580977 19978 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.65.2 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0331 11:13:19.592040 19978 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "8443/tcp") 0).HostPort}}'" old-k8s-version-221000
I0331 11:13:19.654029 19978 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0331 11:13:19.654111 19978 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0331 11:13:19.675697 19978 docker.go:639] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0331 11:13:19.675711 19978 docker.go:645] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0331 11:13:19.675777 19978 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0331 11:13:19.683870 19978 ssh_runner.go:195] Run: which lz4
I0331 11:13:19.688141 19978 ssh_runner.go:195] Run: stat -c "%s %y" /preloaded.tar.lz4
I0331 11:13:19.692172 19978 ssh_runner.go:352] existence check for /preloaded.tar.lz4: stat -c "%s %y" /preloaded.tar.lz4: Process exited with status 1
stdout:
stderr:
stat: cannot stat '/preloaded.tar.lz4': No such file or directory
I0331 11:13:19.692201 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 --> /preloaded.tar.lz4 (369789069 bytes)
I0331 11:13:25.195563 19978 docker.go:603] Took 5.507782 seconds to copy over tarball
I0331 11:13:25.195633 19978 ssh_runner.go:195] Run: sudo tar -I lz4 -C /var -xf /preloaded.tar.lz4
I0331 11:13:27.476808 19978 ssh_runner.go:235] Completed: sudo tar -I lz4 -C /var -xf /preloaded.tar.lz4: (2.281256092s)
I0331 11:13:27.476825 19978 ssh_runner.go:146] rm: /preloaded.tar.lz4
I0331 11:13:27.546246 19978 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0331 11:13:27.554645 19978 ssh_runner.go:362] scp memory --> /var/lib/docker/image/overlay2/repositories.json (2499 bytes)
I0331 11:13:27.567735 19978 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0331 11:13:27.636915 19978 ssh_runner.go:195] Run: sudo systemctl restart docker
I0331 11:13:28.160150 19978 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0331 11:13:28.180747 19978 docker.go:639] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0331 11:13:28.180761 19978 docker.go:645] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0331 11:13:28.180770 19978 cache_images.go:88] LoadImages start: [registry.k8s.io/kube-apiserver:v1.16.0 registry.k8s.io/kube-controller-manager:v1.16.0 registry.k8s.io/kube-scheduler:v1.16.0 registry.k8s.io/kube-proxy:v1.16.0 registry.k8s.io/pause:3.1 registry.k8s.io/etcd:3.3.15-0 registry.k8s.io/coredns:1.6.2 gcr.io/k8s-minikube/storage-provisioner:v5]
I0331 11:13:28.190132 19978 image.go:134] retrieving image: registry.k8s.io/kube-controller-manager:v1.16.0
I0331 11:13:28.192667 19978 image.go:134] retrieving image: registry.k8s.io/pause:3.1
I0331 11:13:28.192816 19978 image.go:134] retrieving image: registry.k8s.io/coredns:1.6.2
I0331 11:13:28.193282 19978 image.go:134] retrieving image: registry.k8s.io/kube-scheduler:v1.16.0
I0331 11:13:28.194760 19978 image.go:134] retrieving image: gcr.io/k8s-minikube/storage-provisioner:v5
I0331 11:13:28.195274 19978 image.go:134] retrieving image: registry.k8s.io/etcd:3.3.15-0
I0331 11:13:28.195778 19978 image.go:134] retrieving image: registry.k8s.io/kube-apiserver:v1.16.0
I0331 11:13:28.199139 19978 image.go:134] retrieving image: registry.k8s.io/kube-proxy:v1.16.0
I0331 11:13:28.204417 19978 image.go:177] daemon lookup for registry.k8s.io/kube-controller-manager:v1.16.0: Error: No such image: registry.k8s.io/kube-controller-manager:v1.16.0
I0331 11:13:28.207759 19978 image.go:177] daemon lookup for registry.k8s.io/coredns:1.6.2: Error: No such image: registry.k8s.io/coredns:1.6.2
I0331 11:13:28.208088 19978 image.go:177] daemon lookup for registry.k8s.io/pause:3.1: Error: No such image: registry.k8s.io/pause:3.1
I0331 11:13:28.209314 19978 image.go:177] daemon lookup for gcr.io/k8s-minikube/storage-provisioner:v5: Error: No such image: gcr.io/k8s-minikube/storage-provisioner:v5
I0331 11:13:28.209422 19978 image.go:177] daemon lookup for registry.k8s.io/kube-scheduler:v1.16.0: Error: No such image: registry.k8s.io/kube-scheduler:v1.16.0
I0331 11:13:28.209963 19978 image.go:177] daemon lookup for registry.k8s.io/etcd:3.3.15-0: Error: No such image: registry.k8s.io/etcd:3.3.15-0
I0331 11:13:28.210200 19978 image.go:177] daemon lookup for registry.k8s.io/kube-apiserver:v1.16.0: Error: No such image: registry.k8s.io/kube-apiserver:v1.16.0
I0331 11:13:28.211571 19978 image.go:177] daemon lookup for registry.k8s.io/kube-proxy:v1.16.0: Error: No such image: registry.k8s.io/kube-proxy:v1.16.0
I0331 11:13:29.360244 19978 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-controller-manager:v1.16.0
I0331 11:13:29.381794 19978 cache_images.go:116] "registry.k8s.io/kube-controller-manager:v1.16.0" needs transfer: "registry.k8s.io/kube-controller-manager:v1.16.0" does not exist at hash "06a629a7e51cdcc81a5ed6a3e6650348312f20c954ac52ee489a023628ec9c7d" in container runtime
I0331 11:13:29.381830 19978 docker.go:313] Removing image: registry.k8s.io/kube-controller-manager:v1.16.0
I0331 11:13:29.381897 19978 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-controller-manager:v1.16.0
I0331 11:13:29.402983 19978 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/16144-2324/.minikube/cache/images/amd64/registry.k8s.io/kube-controller-manager_v1.16.0
I0331 11:13:29.498061 19978 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/coredns:1.6.2
I0331 11:13:29.519511 19978 cache_images.go:116] "registry.k8s.io/coredns:1.6.2" needs transfer: "registry.k8s.io/coredns:1.6.2" does not exist at hash "bf261d157914477ee1a5969d28ec687f3fbfc9fb5a664b22df78e57023b0e03b" in container runtime
I0331 11:13:29.519533 19978 docker.go:313] Removing image: registry.k8s.io/coredns:1.6.2
I0331 11:13:29.519582 19978 ssh_runner.go:195] Run: docker rmi registry.k8s.io/coredns:1.6.2
I0331 11:13:29.541807 19978 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/16144-2324/.minikube/cache/images/amd64/registry.k8s.io/coredns_1.6.2
I0331 11:13:29.634373 19978 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/pause:3.1
I0331 11:13:29.656917 19978 cache_images.go:116] "registry.k8s.io/pause:3.1" needs transfer: "registry.k8s.io/pause:3.1" does not exist at hash "da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e" in container runtime
I0331 11:13:29.656944 19978 docker.go:313] Removing image: registry.k8s.io/pause:3.1
I0331 11:13:29.657002 19978 ssh_runner.go:195] Run: docker rmi registry.k8s.io/pause:3.1
I0331 11:13:29.678742 19978 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/16144-2324/.minikube/cache/images/amd64/registry.k8s.io/pause_3.1
I0331 11:13:30.046458 19978 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-scheduler:v1.16.0
I0331 11:13:30.066665 19978 cache_images.go:116] "registry.k8s.io/kube-scheduler:v1.16.0" needs transfer: "registry.k8s.io/kube-scheduler:v1.16.0" does not exist at hash "301ddc62b80b16315d3c2653cf3888370394277afb3187614cfa20edc352ca0a" in container runtime
I0331 11:13:30.066699 19978 docker.go:313] Removing image: registry.k8s.io/kube-scheduler:v1.16.0
I0331 11:13:30.066757 19978 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-scheduler:v1.16.0
I0331 11:13:30.090444 19978 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/16144-2324/.minikube/cache/images/amd64/registry.k8s.io/kube-scheduler_v1.16.0
I0331 11:13:30.191046 19978 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} gcr.io/k8s-minikube/storage-provisioner:v5
I0331 11:13:30.355795 19978 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/etcd:3.3.15-0
I0331 11:13:30.376480 19978 cache_images.go:116] "registry.k8s.io/etcd:3.3.15-0" needs transfer: "registry.k8s.io/etcd:3.3.15-0" does not exist at hash "b2756210eeabf84f3221da9959e9483f3919dc2aaab4cd45e7cd072fcbde27ed" in container runtime
I0331 11:13:30.376509 19978 docker.go:313] Removing image: registry.k8s.io/etcd:3.3.15-0
I0331 11:13:30.376563 19978 ssh_runner.go:195] Run: docker rmi registry.k8s.io/etcd:3.3.15-0
I0331 11:13:30.397203 19978 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/16144-2324/.minikube/cache/images/amd64/registry.k8s.io/etcd_3.3.15-0
I0331 11:13:30.653506 19978 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-apiserver:v1.16.0
I0331 11:13:30.675335 19978 cache_images.go:116] "registry.k8s.io/kube-apiserver:v1.16.0" needs transfer: "registry.k8s.io/kube-apiserver:v1.16.0" does not exist at hash "b305571ca60a5a7818bda47da122683d75e8a1907475681ee8b1efbd06bff12e" in container runtime
I0331 11:13:30.675360 19978 docker.go:313] Removing image: registry.k8s.io/kube-apiserver:v1.16.0
I0331 11:13:30.675421 19978 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-apiserver:v1.16.0
I0331 11:13:30.698756 19978 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/16144-2324/.minikube/cache/images/amd64/registry.k8s.io/kube-apiserver_v1.16.0
I0331 11:13:30.962395 19978 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-proxy:v1.16.0
I0331 11:13:30.983368 19978 cache_images.go:116] "registry.k8s.io/kube-proxy:v1.16.0" needs transfer: "registry.k8s.io/kube-proxy:v1.16.0" does not exist at hash "c21b0c7400f988db4777858edd13b6d3930d62d7ccf026d2415485a52037f384" in container runtime
I0331 11:13:30.983396 19978 docker.go:313] Removing image: registry.k8s.io/kube-proxy:v1.16.0
I0331 11:13:30.983458 19978 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-proxy:v1.16.0
I0331 11:13:31.015988 19978 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/16144-2324/.minikube/cache/images/amd64/registry.k8s.io/kube-proxy_v1.16.0
I0331 11:13:31.016049 19978 cache_images.go:92] LoadImages completed in 2.835406854s
W0331 11:13:31.016154 19978 out.go:239] X Unable to load cached images: loading cached images: stat /Users/jenkins/minikube-integration/16144-2324/.minikube/cache/images/amd64/registry.k8s.io/kube-controller-manager_v1.16.0: no such file or directory
X Unable to load cached images: loading cached images: stat /Users/jenkins/minikube-integration/16144-2324/.minikube/cache/images/amd64/registry.k8s.io/kube-controller-manager_v1.16.0: no such file or directory
I0331 11:13:31.016243 19978 ssh_runner.go:195] Run: docker info --format {{.CgroupDriver}}
I0331 11:13:31.047295 19978 cni.go:84] Creating CNI manager for ""
I0331 11:13:31.047318 19978 cni.go:161] CNI unnecessary in this configuration, recommending no CNI
I0331 11:13:31.047337 19978 kubeadm.go:87] Using pod CIDR: 10.244.0.0/16
I0331 11:13:31.047352 19978 kubeadm.go:172] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.76.2 APIServerPort:8443 KubernetesVersion:v1.16.0 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:old-k8s-version-221000 NodeName:old-k8s-version-221000 DNSDomain:cluster.local CRISocket:/var/run/dockershim.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.76.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.76.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticP
odPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[hairpinMode:hairpin-veth runtimeRequestTimeout:15m]}
I0331 11:13:31.047459 19978 kubeadm.go:177] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.76.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: "old-k8s-version-221000"
kubeletExtraArgs:
node-ip: 192.168.76.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.76.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: old-k8s-version-221000
controlPlaneEndpoint: control-plane.minikube.internal:8443
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
listen-metrics-urls: http://127.0.0.1:2381,http://192.168.76.2:2381
kubernetesVersion: v1.16.0
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0331 11:13:31.047530 19978 kubeadm.go:968] kubelet [Unit]
Wants=docker.socket
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.16.0/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=old-k8s-version-221000 --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.76.2
[Install]
config:
{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-221000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:}
I0331 11:13:31.047593 19978 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.16.0
I0331 11:13:31.055751 19978 binaries.go:44] Found k8s binaries, skipping transfer
I0331 11:13:31.055808 19978 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0331 11:13:31.063318 19978 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (348 bytes)
I0331 11:13:31.076631 19978 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0331 11:13:31.092394 19978 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2174 bytes)
I0331 11:13:31.109787 19978 ssh_runner.go:195] Run: grep 192.168.76.2 control-plane.minikube.internal$ /etc/hosts
I0331 11:13:31.116134 19978 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.76.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0331 11:13:31.129094 19978 certs.go:56] Setting up /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000 for IP: 192.168.76.2
I0331 11:13:31.129124 19978 certs.go:186] acquiring lock for shared ca certs: {Name:mk1ddc355573fb6044e73c93dd0e9bf4bae32052 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0331 11:13:31.129350 19978 certs.go:195] skipping minikubeCA CA generation: /Users/jenkins/minikube-integration/16144-2324/.minikube/ca.key
I0331 11:13:31.129458 19978 certs.go:195] skipping proxyClientCA CA generation: /Users/jenkins/minikube-integration/16144-2324/.minikube/proxy-client-ca.key
I0331 11:13:31.129524 19978 certs.go:315] generating minikube-user signed cert: /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/client.key
I0331 11:13:31.129554 19978 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/client.crt with IP's: []
I0331 11:13:31.301926 19978 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/client.crt ...
I0331 11:13:31.301948 19978 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/client.crt: {Name:mk79a65773f2eb0c1d433a6848754a5ed4c59021 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0331 11:13:31.330796 19978 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/client.key ...
I0331 11:13:31.330818 19978 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/client.key: {Name:mke63a0164fa0447771c190d589dbe30f6242f74 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0331 11:13:31.352030 19978 certs.go:315] generating minikube signed cert: /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/apiserver.key.31bdca25
I0331 11:13:31.352103 19978 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/apiserver.crt.31bdca25 with IP's: [192.168.76.2 10.96.0.1 127.0.0.1 10.0.0.1]
I0331 11:13:31.412205 19978 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/apiserver.crt.31bdca25 ...
I0331 11:13:31.412224 19978 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/apiserver.crt.31bdca25: {Name:mk38b201eccce6b3d14a3b6c0a5db868e304c7d1 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0331 11:13:31.412527 19978 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/apiserver.key.31bdca25 ...
I0331 11:13:31.412536 19978 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/apiserver.key.31bdca25: {Name:mk9a3f172f130f13f764a93ea61ccf548d0ab6b0 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0331 11:13:31.412748 19978 certs.go:333] copying /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/apiserver.crt.31bdca25 -> /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/apiserver.crt
I0331 11:13:31.412924 19978 certs.go:337] copying /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/apiserver.key.31bdca25 -> /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/apiserver.key
I0331 11:13:31.413094 19978 certs.go:315] generating aggregator signed cert: /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/proxy-client.key
I0331 11:13:31.413111 19978 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/proxy-client.crt with IP's: []
I0331 11:13:31.575834 19978 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/proxy-client.crt ...
I0331 11:13:31.575849 19978 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/proxy-client.crt: {Name:mka6d12b78bc0da19b38546fca36e937d1506199 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0331 11:13:31.576160 19978 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/proxy-client.key ...
I0331 11:13:31.576168 19978 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/proxy-client.key: {Name:mkbf1a71690781dee25fa1b26ff4804210864793 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0331 11:13:31.576562 19978 certs.go:401] found cert: /Users/jenkins/minikube-integration/16144-2324/.minikube/certs/Users/jenkins/minikube-integration/16144-2324/.minikube/certs/2800.pem (1338 bytes)
W0331 11:13:31.576607 19978 certs.go:397] ignoring /Users/jenkins/minikube-integration/16144-2324/.minikube/certs/Users/jenkins/minikube-integration/16144-2324/.minikube/certs/2800_empty.pem, impossibly tiny 0 bytes
I0331 11:13:31.576643 19978 certs.go:401] found cert: /Users/jenkins/minikube-integration/16144-2324/.minikube/certs/Users/jenkins/minikube-integration/16144-2324/.minikube/certs/ca-key.pem (1679 bytes)
I0331 11:13:31.576685 19978 certs.go:401] found cert: /Users/jenkins/minikube-integration/16144-2324/.minikube/certs/Users/jenkins/minikube-integration/16144-2324/.minikube/certs/ca.pem (1078 bytes)
I0331 11:13:31.576717 19978 certs.go:401] found cert: /Users/jenkins/minikube-integration/16144-2324/.minikube/certs/Users/jenkins/minikube-integration/16144-2324/.minikube/certs/cert.pem (1123 bytes)
I0331 11:13:31.576754 19978 certs.go:401] found cert: /Users/jenkins/minikube-integration/16144-2324/.minikube/certs/Users/jenkins/minikube-integration/16144-2324/.minikube/certs/key.pem (1679 bytes)
I0331 11:13:31.576835 19978 certs.go:401] found cert: /Users/jenkins/minikube-integration/16144-2324/.minikube/files/etc/ssl/certs/Users/jenkins/minikube-integration/16144-2324/.minikube/files/etc/ssl/certs/28002.pem (1708 bytes)
I0331 11:13:31.577307 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1399 bytes)
I0331 11:13:31.601956 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1679 bytes)
I0331 11:13:31.624680 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0331 11:13:31.647272 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/profiles/old-k8s-version-221000/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1679 bytes)
I0331 11:13:31.664947 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0331 11:13:31.683044 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1679 bytes)
I0331 11:13:31.708272 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0331 11:13:31.728908 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1679 bytes)
I0331 11:13:31.749870 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/files/etc/ssl/certs/28002.pem --> /usr/share/ca-certificates/28002.pem (1708 bytes)
I0331 11:13:31.767822 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0331 11:13:31.786381 19978 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16144-2324/.minikube/certs/2800.pem --> /usr/share/ca-certificates/2800.pem (1338 bytes)
I0331 11:13:31.817569 19978 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0331 11:13:31.833503 19978 ssh_runner.go:195] Run: openssl version
I0331 11:13:31.839498 19978 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0331 11:13:31.847710 19978 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0331 11:13:31.851811 19978 certs.go:444] hashing: -rw-r--r-- 1 root root 1111 Mar 31 17:21 /usr/share/ca-certificates/minikubeCA.pem
I0331 11:13:31.851854 19978 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0331 11:13:31.857398 19978 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0331 11:13:31.865782 19978 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/2800.pem && ln -fs /usr/share/ca-certificates/2800.pem /etc/ssl/certs/2800.pem"
I0331 11:13:31.874089 19978 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/2800.pem
I0331 11:13:31.878301 19978 certs.go:444] hashing: -rw-r--r-- 1 root root 1338 Mar 31 17:26 /usr/share/ca-certificates/2800.pem
I0331 11:13:31.878345 19978 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/2800.pem
I0331 11:13:31.884009 19978 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/2800.pem /etc/ssl/certs/51391683.0"
I0331 11:13:31.893738 19978 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/28002.pem && ln -fs /usr/share/ca-certificates/28002.pem /etc/ssl/certs/28002.pem"
I0331 11:13:31.904832 19978 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/28002.pem
I0331 11:13:31.910244 19978 certs.go:444] hashing: -rw-r--r-- 1 root root 1708 Mar 31 17:26 /usr/share/ca-certificates/28002.pem
I0331 11:13:31.910321 19978 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/28002.pem
I0331 11:13:31.916767 19978 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/28002.pem /etc/ssl/certs/3ec20f2e.0"
I0331 11:13:31.928495 19978 kubeadm.go:401] StartCluster: {Name:old-k8s-version-221000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.38@sha256:516db0892e1cd79b6781fc1a102fca4bf392576bbf3ca0fa01a467cb6cc0af55 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-221000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs
:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQe
muFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP:}
I0331 11:13:31.928614 19978 ssh_runner.go:195] Run: docker ps --filter status=paused --filter=name=k8s_.*_(kube-system)_ --format={{.ID}}
I0331 11:13:31.951156 19978 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0331 11:13:31.959289 19978 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0331 11:13:31.966801 19978 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0331 11:13:31.966857 19978 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0331 11:13:31.974355 19978 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0331 11:13:31.974383 19978 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0331 11:13:32.045003 19978 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0331 11:13:32.045043 19978 kubeadm.go:322] [preflight] Running pre-flight checks
I0331 11:13:32.241796 19978 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0331 11:13:32.241886 19978 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0331 11:13:32.241952 19978 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0331 11:13:32.412546 19978 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0331 11:13:32.413256 19978 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0331 11:13:32.420741 19978 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0331 11:13:32.489688 19978 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0331 11:13:32.610916 19978 out.go:204] - Generating certificates and keys ...
I0331 11:13:32.611005 19978 kubeadm.go:322] [certs] Using existing ca certificate authority
I0331 11:13:32.611070 19978 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0331 11:13:32.763241 19978 kubeadm.go:322] [certs] Generating "apiserver-kubelet-client" certificate and key
I0331 11:13:32.877876 19978 kubeadm.go:322] [certs] Generating "front-proxy-ca" certificate and key
I0331 11:13:33.074958 19978 kubeadm.go:322] [certs] Generating "front-proxy-client" certificate and key
I0331 11:13:33.139253 19978 kubeadm.go:322] [certs] Generating "etcd/ca" certificate and key
I0331 11:13:33.289102 19978 kubeadm.go:322] [certs] Generating "etcd/server" certificate and key
I0331 11:13:33.289206 19978 kubeadm.go:322] [certs] etcd/server serving cert is signed for DNS names [old-k8s-version-221000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
I0331 11:13:33.385744 19978 kubeadm.go:322] [certs] Generating "etcd/peer" certificate and key
I0331 11:13:33.385848 19978 kubeadm.go:322] [certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-221000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
I0331 11:13:33.483327 19978 kubeadm.go:322] [certs] Generating "etcd/healthcheck-client" certificate and key
I0331 11:13:33.609083 19978 kubeadm.go:322] [certs] Generating "apiserver-etcd-client" certificate and key
I0331 11:13:33.689936 19978 kubeadm.go:322] [certs] Generating "sa" key and public key
I0331 11:13:33.690145 19978 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0331 11:13:33.806239 19978 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0331 11:13:33.872452 19978 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0331 11:13:33.922647 19978 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0331 11:13:34.101524 19978 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0331 11:13:34.102032 19978 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0331 11:13:34.123666 19978 out.go:204] - Booting up control plane ...
I0331 11:13:34.123765 19978 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0331 11:13:34.123831 19978 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0331 11:13:34.123893 19978 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0331 11:13:34.123971 19978 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0331 11:13:34.124131 19978 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0331 11:14:14.108462 19978 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0331 11:14:14.109268 19978 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0331 11:14:14.109483 19978 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0331 11:14:19.109928 19978 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0331 11:14:19.110104 19978 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0331 11:14:29.110070 19978 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0331 11:14:29.110223 19978 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0331 11:14:49.109671 19978 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0331 11:14:49.109894 19978 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0331 11:15:29.108400 19978 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0331 11:15:29.108604 19978 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0331 11:15:29.108612 19978 kubeadm.go:322]
I0331 11:15:29.108648 19978 kubeadm.go:322] Unfortunately, an error has occurred:
I0331 11:15:29.108687 19978 kubeadm.go:322] timed out waiting for the condition
I0331 11:15:29.108699 19978 kubeadm.go:322]
I0331 11:15:29.108727 19978 kubeadm.go:322] This error is likely caused by:
I0331 11:15:29.108762 19978 kubeadm.go:322] - The kubelet is not running
I0331 11:15:29.108871 19978 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0331 11:15:29.108903 19978 kubeadm.go:322]
I0331 11:15:29.109022 19978 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0331 11:15:29.109075 19978 kubeadm.go:322] - 'systemctl status kubelet'
I0331 11:15:29.109102 19978 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0331 11:15:29.109106 19978 kubeadm.go:322]
I0331 11:15:29.109237 19978 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0331 11:15:29.109320 19978 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0331 11:15:29.109393 19978 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0331 11:15:29.109433 19978 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0331 11:15:29.109496 19978 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0331 11:15:29.109530 19978 kubeadm.go:322] - 'docker logs CONTAINERID'
I0331 11:15:29.112728 19978 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0331 11:15:29.112840 19978 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0331 11:15:29.112974 19978 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.2. Latest validated version: 18.09
I0331 11:15:29.113104 19978 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0331 11:15:29.113193 19978 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0331 11:15:29.113251 19978 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
W0331 11:15:29.113464 19978 out.go:239] ! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-221000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-221000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.2. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-221000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-221000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.2. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
I0331 11:15:29.113513 19978 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm reset --cri-socket /var/run/dockershim.sock --force"
I0331 11:15:29.528497 19978 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service kubelet
I0331 11:15:29.538728 19978 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0331 11:15:29.538784 19978 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0331 11:15:29.546737 19978 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0331 11:15:29.546786 19978 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0331 11:15:29.596006 19978 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0331 11:15:29.596056 19978 kubeadm.go:322] [preflight] Running pre-flight checks
I0331 11:15:29.766988 19978 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0331 11:15:29.767078 19978 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0331 11:15:29.767201 19978 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0331 11:15:29.920754 19978 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0331 11:15:29.921399 19978 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0331 11:15:29.929353 19978 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0331 11:15:30.011024 19978 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0331 11:15:30.068267 19978 out.go:204] - Generating certificates and keys ...
I0331 11:15:30.068363 19978 kubeadm.go:322] [certs] Using existing ca certificate authority
I0331 11:15:30.068472 19978 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0331 11:15:30.068582 19978 kubeadm.go:322] [certs] Using existing apiserver-kubelet-client certificate and key on disk
I0331 11:15:30.068671 19978 kubeadm.go:322] [certs] Using existing front-proxy-ca certificate authority
I0331 11:15:30.068748 19978 kubeadm.go:322] [certs] Using existing front-proxy-client certificate and key on disk
I0331 11:15:30.068793 19978 kubeadm.go:322] [certs] Using existing etcd/ca certificate authority
I0331 11:15:30.068838 19978 kubeadm.go:322] [certs] Using existing etcd/server certificate and key on disk
I0331 11:15:30.068887 19978 kubeadm.go:322] [certs] Using existing etcd/peer certificate and key on disk
I0331 11:15:30.068957 19978 kubeadm.go:322] [certs] Using existing etcd/healthcheck-client certificate and key on disk
I0331 11:15:30.069009 19978 kubeadm.go:322] [certs] Using existing apiserver-etcd-client certificate and key on disk
I0331 11:15:30.069037 19978 kubeadm.go:322] [certs] Using the existing "sa" key
I0331 11:15:30.069084 19978 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0331 11:15:30.163457 19978 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0331 11:15:30.285277 19978 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0331 11:15:30.405130 19978 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0331 11:15:30.558077 19978 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0331 11:15:30.559207 19978 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0331 11:15:30.580868 19978 out.go:204] - Booting up control plane ...
I0331 11:15:30.581055 19978 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0331 11:15:30.581229 19978 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0331 11:15:30.581359 19978 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0331 11:15:30.581496 19978 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0331 11:15:30.581812 19978 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0331 11:16:10.569607 19978 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0331 11:16:10.570289 19978 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0331 11:16:10.570550 19978 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0331 11:16:15.579280 19978 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0331 11:16:15.579459 19978 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0331 11:16:25.591018 19978 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0331 11:16:25.591230 19978 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0331 11:16:45.598653 19978 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0331 11:16:45.598812 19978 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0331 11:17:25.600416 19978 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0331 11:17:25.600566 19978 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0331 11:17:25.600580 19978 kubeadm.go:322]
I0331 11:17:25.600622 19978 kubeadm.go:322] Unfortunately, an error has occurred:
I0331 11:17:25.600656 19978 kubeadm.go:322] timed out waiting for the condition
I0331 11:17:25.600660 19978 kubeadm.go:322]
I0331 11:17:25.600688 19978 kubeadm.go:322] This error is likely caused by:
I0331 11:17:25.600721 19978 kubeadm.go:322] - The kubelet is not running
I0331 11:17:25.600813 19978 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0331 11:17:25.600819 19978 kubeadm.go:322]
I0331 11:17:25.600908 19978 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0331 11:17:25.600940 19978 kubeadm.go:322] - 'systemctl status kubelet'
I0331 11:17:25.600973 19978 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0331 11:17:25.600981 19978 kubeadm.go:322]
I0331 11:17:25.601075 19978 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0331 11:17:25.601154 19978 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0331 11:17:25.601227 19978 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0331 11:17:25.601270 19978 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0331 11:17:25.601339 19978 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0331 11:17:25.601370 19978 kubeadm.go:322] - 'docker logs CONTAINERID'
I0331 11:17:25.604396 19978 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0331 11:17:25.604466 19978 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0331 11:17:25.604566 19978 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.2. Latest validated version: 18.09
I0331 11:17:25.604657 19978 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0331 11:17:25.604743 19978 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0331 11:17:25.604803 19978 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
I0331 11:17:25.604828 19978 kubeadm.go:403] StartCluster complete in 3m53.654846206s
I0331 11:17:25.604921 19978 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-apiserver --format={{.ID}}
I0331 11:17:25.624430 19978 logs.go:277] 0 containers: []
W0331 11:17:25.624444 19978 logs.go:279] No container was found matching "kube-apiserver"
I0331 11:17:25.624513 19978 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_etcd --format={{.ID}}
I0331 11:17:25.644797 19978 logs.go:277] 0 containers: []
W0331 11:17:25.644810 19978 logs.go:279] No container was found matching "etcd"
I0331 11:17:25.644883 19978 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_coredns --format={{.ID}}
I0331 11:17:25.663538 19978 logs.go:277] 0 containers: []
W0331 11:17:25.663552 19978 logs.go:279] No container was found matching "coredns"
I0331 11:17:25.663619 19978 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-scheduler --format={{.ID}}
I0331 11:17:25.682601 19978 logs.go:277] 0 containers: []
W0331 11:17:25.682614 19978 logs.go:279] No container was found matching "kube-scheduler"
I0331 11:17:25.682689 19978 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-proxy --format={{.ID}}
I0331 11:17:25.702364 19978 logs.go:277] 0 containers: []
W0331 11:17:25.702376 19978 logs.go:279] No container was found matching "kube-proxy"
I0331 11:17:25.702447 19978 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-controller-manager --format={{.ID}}
I0331 11:17:25.721447 19978 logs.go:277] 0 containers: []
W0331 11:17:25.721460 19978 logs.go:279] No container was found matching "kube-controller-manager"
I0331 11:17:25.721528 19978 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kindnet --format={{.ID}}
I0331 11:17:25.742384 19978 logs.go:277] 0 containers: []
W0331 11:17:25.742398 19978 logs.go:279] No container was found matching "kindnet"
I0331 11:17:25.742405 19978 logs.go:123] Gathering logs for kubelet ...
I0331 11:17:25.742416 19978 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
I0331 11:17:25.780658 19978 logs.go:123] Gathering logs for dmesg ...
I0331 11:17:25.780690 19978 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0331 11:17:25.793929 19978 logs.go:123] Gathering logs for describe nodes ...
I0331 11:17:25.793943 19978 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0331 11:17:25.850793 19978 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0331 11:17:25.850805 19978 logs.go:123] Gathering logs for Docker ...
I0331 11:17:25.850812 19978 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u docker -u cri-docker -n 400"
I0331 11:17:25.882443 19978 logs.go:123] Gathering logs for container status ...
I0331 11:17:25.882461 19978 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0331 11:17:27.931585 19978 ssh_runner.go:235] Completed: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a": (2.04918268s)
W0331 11:17:27.931720 19978 out.go:369] Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.2. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0331 11:17:27.931739 19978 out.go:239] *
*
W0331 11:17:27.931848 19978 out.go:239] X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.2. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.2. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0331 11:17:27.931861 19978 out.go:239] *
*
W0331 11:17:27.932481 19978 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0331 11:17:28.020341 19978 out.go:177]
W0331 11:17:28.083390 19978 out.go:239] X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.2. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.2. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0331 11:17:28.083517 19978 out.go:239] * Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
* Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
W0331 11:17:28.083624 19978 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/4172
* Related issue: https://github.com/kubernetes/minikube/issues/4172
I0331 11:17:28.125074 19978 out.go:177]
** /stderr **
start_stop_delete_test.go:188: failed starting minikube -first start-. args "out/minikube-darwin-amd64 start -p old-k8s-version-221000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0": exit status 109
helpers_test.go:222: -----------------------post-mortem--------------------------------
helpers_test.go:230: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/FirstStart]: docker inspect <======
helpers_test.go:231: (dbg) Run: docker inspect old-k8s-version-221000
helpers_test.go:235: (dbg) docker inspect old-k8s-version-221000:
-- stdout --
[
{
"Id": "0bb0a05e14047e05df95551f0074a3f51840d9971494dab21229cc308c9d620c",
"Created": "2023-03-31T18:13:14.794492262Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 278398,
"ExitCode": 0,
"Error": "",
"StartedAt": "2023-03-31T18:13:15.104874322Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:e2a21e2966a9bc54932b0177ccaaf147775c28fd6729fa50fc93f998eb5d1d4e",
"ResolvConfPath": "/var/lib/docker/containers/0bb0a05e14047e05df95551f0074a3f51840d9971494dab21229cc308c9d620c/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/0bb0a05e14047e05df95551f0074a3f51840d9971494dab21229cc308c9d620c/hostname",
"HostsPath": "/var/lib/docker/containers/0bb0a05e14047e05df95551f0074a3f51840d9971494dab21229cc308c9d620c/hosts",
"LogPath": "/var/lib/docker/containers/0bb0a05e14047e05df95551f0074a3f51840d9971494dab21229cc308c9d620c/0bb0a05e14047e05df95551f0074a3f51840d9971494dab21229cc308c9d620c-json.log",
"Name": "/old-k8s-version-221000",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro",
"old-k8s-version-221000:/var"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "old-k8s-version-221000",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 2306867200,
"NanoCpus": 2000000000,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 2306867200,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/9371145efd9e65e4cf9bb0a81f4b673e60c5dc231a80c5de8008817807bc8756-init/diff:/var/lib/docker/overlay2/c52de480a9d3b92156a6b7f40b9f29c89c00bff0dc7d6acec95d15bf8fa7e706/diff:/var/lib/docker/overlay2/4fa51510fbaca99d18589345b5c49ac647c3852526857e140385c8c74142d864/diff:/var/lib/docker/overlay2/cf9c75d0f98b371f5655e6f7a9422b077615b807b5ded6caad5cb3ade54a6bcf/diff:/var/lib/docker/overlay2/b5f4d681c4091990a5aedc7eba843be0d1f3bb91c8805b248e118c1a15aeb426/diff:/var/lib/docker/overlay2/726f8f99c4617031c8c79d131e446f30d959a0783880b92dd26264e6e07f578f/diff:/var/lib/docker/overlay2/008245a012fc592a94495b269e28d4d957b15a5f74e01a1fcbf876c5a4ba70d1/diff:/var/lib/docker/overlay2/e814b23972aa1481aab63bf91bb25741253bd9f081c67374e1f699c38c83e20b/diff:/var/lib/docker/overlay2/c78d407937cce04bc3c30a83fbf1b7d21b115b59f4095d751a209b86004e5084/diff:/var/lib/docker/overlay2/ee5f9401c2be285db119edbae2a99aed2dcb647e8d11cf47fca0347187d62e4b/diff:/var/lib/docker/overlay2/332be6
c4fb796c3c6b37a76dafc41ec8a1ba8e959fd7d71a94f827b6fb735ad4/diff:/var/lib/docker/overlay2/d6b9e54e2d5bb577d56e176337be5c5b76cd0af24af6644eb07287be261db26b/diff:/var/lib/docker/overlay2/2d0494381df049a5d91fa5ab305c5e51b253d3e85b0218e811be4f8356a37428/diff:/var/lib/docker/overlay2/afd882240733a5ed95e43e2142b7cb8a2b4d1326880618cc2324b03915020c4c/diff:/var/lib/docker/overlay2/33fa3700a4a4c49f289c610638ccb45ed575386bfb3064629f04c300421c4310/diff:/var/lib/docker/overlay2/e98659a1347b2114201116e62517d120bf4e0142318c89985a118ad2ae3e26a1/diff:/var/lib/docker/overlay2/5c7c035c89c1bcce5e2168402e485f3512a039670e3d6dd9fc3d76fb08f8244a/diff:/var/lib/docker/overlay2/ef4639b18525d48d115761fc8c9f0e9a4a49d9b1e2fe1dee9e17693200e24e74/diff:/var/lib/docker/overlay2/94b8ff4f6f12f9180a3bd2f3938f4cb57670fbb53250a7c3dbf644d1d1e6dea1/diff:/var/lib/docker/overlay2/58bf45edc67803e952c718a39796c16083117bdb5cf686e2f5854a023396b032/diff:/var/lib/docker/overlay2/822a54dff24a23d74321b1adf2e843efd31f58b4cedd73f9b2e3475250134d45/diff:/var/lib/d
ocker/overlay2/ee6c9457f9446cba6d6a0f198210c8beed12156fcdd969bc56f17417e918807d/diff:/var/lib/docker/overlay2/f95430109fc5db985ed6ca29cf75f665a17355890956c738d6d95c768cfbf69a/diff:/var/lib/docker/overlay2/cc22b7f9ebaea7002093337d464d55be8275142d31159d9ebdec3a9850a5f950/diff:/var/lib/docker/overlay2/b160c09d12608cec7a0042efb55263ffbdcb36ec0b0d73018e4cb4d726408d81/diff:/var/lib/docker/overlay2/5f6ff7fa8d97499d42cbb31528fe37f008f703abfcbbe973ed0d9f145c9d6039/diff:/var/lib/docker/overlay2/fabd7993133916925eded9ca46e7df8102d62c181ab0c081245d7c1ff1283c27/diff:/var/lib/docker/overlay2/2e6ba7aa5cb90faf1e555f4e520096483fbd232af03f3692ac51612714d0e385/diff:/var/lib/docker/overlay2/3a9104d80fb41426d356ca9e7fa94d0985824ceed9552e14890f18baccb9efa3/diff:/var/lib/docker/overlay2/9fb0d3a7b4b028d223f98735c60cf8066a223c50c202ee97ebe9d34a53f2513c/diff:/var/lib/docker/overlay2/06e8638a1f85e84a5044d94c8f64c3db2e4d2ec069d74632686067d7bb4b5172/diff:/var/lib/docker/overlay2/b986e2ce1a3377c222863b76fcfc811e9f7f3af845fae4a031c1be7034a
2db30/diff:/var/lib/docker/overlay2/8c61ff71163863f677f6c1cf8517ea53d543086afddf87fbcad9200e3d175b61/diff:/var/lib/docker/overlay2/5ce942778f0cdc742635ab8f4ee5aec345051ff4b67d4195e6aaa66c4aae4e14/diff:/var/lib/docker/overlay2/7d0a926a2580ecaf6b2aead105bb64b77d4837d3ea6e0c85cd95fdd3333f00bb/diff:/var/lib/docker/overlay2/d0d03db4cef8aaf8782b17d0626ad44f733ce7f04c3e21bf65084b97c6ea67cb/diff:/var/lib/docker/overlay2/e883b9d6436927d753216e62fe82d039db2f23ddcf499bf20a314e3430f7daef/diff:/var/lib/docker/overlay2/7718289812bceacd3143fbe5fd71a56482d7c577a981b237e8c007ff52731628/diff:/var/lib/docker/overlay2/acc3c766512d89dd02b3d3b06cbb9c7967ed29e4c8153cd9619d018c6b6de87a/diff:/var/lib/docker/overlay2/c7cab844f64ef8e73212fc5acb293faa5813da467c3228c985f682ec2cb30164/diff:/var/lib/docker/overlay2/753578deafc777ffa7c6d2a91835e011b17cb8336d09ba1adda245eccf3fbe12/diff:/var/lib/docker/overlay2/6a8e8d84668fb5b2eb3554aeb439a789e0efad3dde621d850c5c522256ea168c/diff:/var/lib/docker/overlay2/73847ab62012310cb9c6b55b335aa966ece312
b33e0dc4c7be39ab7733b4f1ac/diff",
"MergedDir": "/var/lib/docker/overlay2/9371145efd9e65e4cf9bb0a81f4b673e60c5dc231a80c5de8008817807bc8756/merged",
"UpperDir": "/var/lib/docker/overlay2/9371145efd9e65e4cf9bb0a81f4b673e60c5dc231a80c5de8008817807bc8756/diff",
"WorkDir": "/var/lib/docker/overlay2/9371145efd9e65e4cf9bb0a81f4b673e60c5dc231a80c5de8008817807bc8756/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "volume",
"Name": "old-k8s-version-221000",
"Source": "/var/lib/docker/volumes/old-k8s-version-221000/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
},
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "old-k8s-version-221000",
"Domainname": "",
"User": "root",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"22/tcp": {},
"2376/tcp": {},
"32443/tcp": {},
"5000/tcp": {},
"8443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase:v0.0.38@sha256:516db0892e1cd79b6781fc1a102fca4bf392576bbf3ca0fa01a467cb6cc0af55",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "old-k8s-version-221000",
"name.minikube.sigs.k8s.io": "old-k8s-version-221000",
"org.opencontainers.image.ref.name": "ubuntu",
"org.opencontainers.image.version": "20.04",
"role.minikube.sigs.k8s.io": ""
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "1435af2e140a84e1562075c16fcbb65a3e0ccdee2aaf0c14ae6d1b2df689a153",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "53352"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "53353"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "53354"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "53355"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "53351"
}
]
},
"SandboxKey": "/var/run/docker/netns/1435af2e140a",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"old-k8s-version-221000": {
"IPAMConfig": {
"IPv4Address": "192.168.76.2"
},
"Links": null,
"Aliases": [
"0bb0a05e1404",
"old-k8s-version-221000"
],
"NetworkID": "1369008204ce2a861d531490c08c0f4f11e7797b90e56bf4d65905b433bee06b",
"EndpointID": "f2d42d9a2ab7f90bacf6faa8fc853efa0d48bf0f1b13814af4a6d84f9440be5a",
"Gateway": "192.168.76.1",
"IPAddress": "192.168.76.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:c0:a8:4c:02",
"DriverOpts": null
}
}
}
}
]
-- /stdout --
helpers_test.go:239: (dbg) Run: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-221000 -n old-k8s-version-221000
helpers_test.go:239: (dbg) Non-zero exit: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-221000 -n old-k8s-version-221000: exit status 6 (409.573151ms)
-- stdout --
Running
WARNING: Your kubectl is pointing to stale minikube-vm.
To fix the kubectl context, run `minikube update-context`
-- /stdout --
** stderr **
E0331 11:17:28.672531 21244 status.go:415] kubeconfig endpoint: extract IP: "old-k8s-version-221000" does not appear in /Users/jenkins/minikube-integration/16144-2324/kubeconfig
** /stderr **
helpers_test.go:239: status error: exit status 6 (may be ok)
helpers_test.go:241: "old-k8s-version-221000" host is not running, skipping log retrieval (state="Running\nWARNING: Your kubectl is pointing to stale minikube-vm.\nTo fix the kubectl context, run `minikube update-context`")
--- FAIL: TestStartStop/group/old-k8s-version/serial/FirstStart (262.34s)