=== RUN TestStartStop/group/old-k8s-version/serial/FirstStart
start_stop_delete_test.go:186: (dbg) Run: out/minikube-darwin-amd64 start -p old-k8s-version-331000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0
E0330 09:29:18.553513 25448 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/kindnet-378000/client.crt: no such file or directory
E0330 09:29:23.674163 25448 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/kindnet-378000/client.crt: no such file or directory
E0330 09:29:33.981775 25448 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/kindnet-378000/client.crt: no such file or directory
start_stop_delete_test.go:186: (dbg) Non-zero exit: out/minikube-darwin-amd64 start -p old-k8s-version-331000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0: exit status 109 (4m12.000413301s)
-- stdout --
* [old-k8s-version-331000] minikube v1.29.0 on Darwin 13.3
- MINIKUBE_LOCATION=16199
- KUBECONFIG=/Users/jenkins/minikube-integration/16199-24978/kubeconfig
- MINIKUBE_BIN=out/minikube-darwin-amd64
- MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
- MINIKUBE_HOME=/Users/jenkins/minikube-integration/16199-24978/.minikube
- MINIKUBE_FORCE_SYSTEMD=
* Using the docker driver based on user configuration
* Using Docker Desktop driver with root privileges
* Starting control plane node old-k8s-version-331000 in cluster old-k8s-version-331000
* Pulling base image ...
* Creating docker container (CPUs=2, Memory=2200MB) ...
* Preparing Kubernetes v1.16.0 on Docker 23.0.1 ...
- Generating certificates and keys ...
- Booting up control plane ...
- Generating certificates and keys ...
- Booting up control plane ...
-- /stdout --
** stderr **
I0330 09:29:17.245896 42664 out.go:296] Setting OutFile to fd 1 ...
I0330 09:29:17.246182 42664 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0330 09:29:17.246188 42664 out.go:309] Setting ErrFile to fd 2...
I0330 09:29:17.246193 42664 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0330 09:29:17.246326 42664 root.go:336] Updating PATH: /Users/jenkins/minikube-integration/16199-24978/.minikube/bin
I0330 09:29:17.248869 42664 out.go:303] Setting JSON to false
I0330 09:29:17.272763 42664 start.go:125] hostinfo: {"hostname":"MacOS-Agent-3.local","uptime":8925,"bootTime":1680184832,"procs":433,"os":"darwin","platform":"darwin","platformFamily":"Standalone Workstation","platformVersion":"13.3","kernelVersion":"22.4.0","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"bd1c05a8-24a6-5973-aa69-f3c7c66a87ce"}
W0330 09:29:17.272875 42664 start.go:133] gopshost.Virtualization returned error: not implemented yet
I0330 09:29:17.294473 42664 out.go:177] * [old-k8s-version-331000] minikube v1.29.0 on Darwin 13.3
I0330 09:29:17.336590 42664 notify.go:220] Checking for updates...
I0330 09:29:17.357514 42664 out.go:177] - MINIKUBE_LOCATION=16199
I0330 09:29:17.378546 42664 out.go:177] - KUBECONFIG=/Users/jenkins/minikube-integration/16199-24978/kubeconfig
I0330 09:29:17.399494 42664 out.go:177] - MINIKUBE_BIN=out/minikube-darwin-amd64
I0330 09:29:17.420561 42664 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0330 09:29:17.441412 42664 out.go:177] - MINIKUBE_HOME=/Users/jenkins/minikube-integration/16199-24978/.minikube
I0330 09:29:17.483482 42664 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0330 09:29:17.504696 42664 config.go:182] Loaded profile config "kubenet-378000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.26.3
I0330 09:29:17.504753 42664 driver.go:365] Setting default libvirt URI to qemu:///system
I0330 09:29:17.588450 42664 docker.go:121] docker version: linux-20.10.23:Docker Desktop 4.17.0 (99724)
I0330 09:29:17.588561 42664 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0330 09:29:17.799774 42664 info.go:266] docker info: {ID:KGYP:ZLCS:CEMV:WBWD:NZSP:U3XB:GRL6:KWCU:PIJM:P62J:I6PQ:YUNG Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:57 OomKillDisable:false NGoroutines:56 SystemTime:2023-03-30 16:29:17.643084263 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:5 KernelVersion:5.15.49-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServ
erAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:6 MemTotal:6231715840 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:20.10.23 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:2456e983eb9e37e47538f59ea18f2043c9a73640 Expected:2456e983eb9e37e47538f59ea18f2043c9a73640} RuncCommit:{ID:v1.1.4-0-g5fd4c4d Expected:v1.1.4-0-g5fd4c4d} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=defaul
t name=cgroupns] ProductLicense: Warnings:<nil> ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.10.3] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.15.1] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker Dev Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Manages Docker extensions Vendor:Docker Inc. Version:v0.2.18] map[Name:sbom Path
:/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM) for an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scan] ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.25.0] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Command line tool for Docker Scout Vendor:Docker Inc. Version:v0.6.0]] Warnings:<nil>}}
I0330 09:29:17.821989 42664 out.go:177] * Using the docker driver based on user configuration
I0330 09:29:17.864392 42664 start.go:295] selected driver: docker
I0330 09:29:17.864403 42664 start.go:859] validating driver "docker" against <nil>
I0330 09:29:17.864411 42664 start.go:870] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0330 09:29:17.867275 42664 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0330 09:29:18.069612 42664 info.go:266] docker info: {ID:KGYP:ZLCS:CEMV:WBWD:NZSP:U3XB:GRL6:KWCU:PIJM:P62J:I6PQ:YUNG Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:57 OomKillDisable:false NGoroutines:55 SystemTime:2023-03-30 16:29:17.923508705 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:5 KernelVersion:5.15.49-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServ
erAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:6 MemTotal:6231715840 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:20.10.23 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:2456e983eb9e37e47538f59ea18f2043c9a73640 Expected:2456e983eb9e37e47538f59ea18f2043c9a73640} RuncCommit:{ID:v1.1.4-0-g5fd4c4d Expected:v1.1.4-0-g5fd4c4d} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=defaul
t name=cgroupns] ProductLicense: Warnings:<nil> ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.10.3] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.15.1] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker Dev Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Manages Docker extensions Vendor:Docker Inc. Version:v0.2.18] map[Name:sbom Path
:/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM) for an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scan] ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.25.0] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Command line tool for Docker Scout Vendor:Docker Inc. Version:v0.6.0]] Warnings:<nil>}}
I0330 09:29:18.069743 42664 start_flags.go:305] no existing cluster config was found, will generate one from the flags
I0330 09:29:18.069932 42664 start_flags.go:919] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0330 09:29:18.091935 42664 out.go:177] * Using Docker Desktop driver with root privileges
I0330 09:29:18.112790 42664 cni.go:84] Creating CNI manager for ""
I0330 09:29:18.112829 42664 cni.go:161] CNI unnecessary in this configuration, recommending no CNI
I0330 09:29:18.112840 42664 start_flags.go:319] config:
{Name:old-k8s-version-331000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.37-1679075007-16079@sha256:9d2bf610e530f47c7e43df9719717075e4c7ec510aea6047b29208bffe7f1978 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-331000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local Containe
rRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP:}
I0330 09:29:18.134661 42664 out.go:177] * Starting control plane node old-k8s-version-331000 in cluster old-k8s-version-331000
I0330 09:29:18.155618 42664 cache.go:120] Beginning downloading kic base image for docker with docker
I0330 09:29:18.176406 42664 out.go:177] * Pulling base image ...
I0330 09:29:18.197621 42664 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0330 09:29:18.197649 42664 image.go:79] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.37-1679075007-16079@sha256:9d2bf610e530f47c7e43df9719717075e4c7ec510aea6047b29208bffe7f1978 in local docker daemon
I0330 09:29:18.197681 42664 preload.go:148] Found local preload: /Users/jenkins/minikube-integration/16199-24978/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4
I0330 09:29:18.197691 42664 cache.go:57] Caching tarball of preloaded images
I0330 09:29:18.197830 42664 preload.go:174] Found /Users/jenkins/minikube-integration/16199-24978/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0330 09:29:18.197842 42664 cache.go:60] Finished verifying existence of preloaded tar for v1.16.0 on docker
I0330 09:29:18.198412 42664 profile.go:148] Saving config to /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/config.json ...
I0330 09:29:18.198505 42664 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/config.json: {Name:mk60ffde538431f22a91f0eac7a1f4d1745248d1 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0330 09:29:18.260626 42664 image.go:83] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.37-1679075007-16079@sha256:9d2bf610e530f47c7e43df9719717075e4c7ec510aea6047b29208bffe7f1978 in local docker daemon, skipping pull
I0330 09:29:18.260647 42664 cache.go:143] gcr.io/k8s-minikube/kicbase-builds:v0.0.37-1679075007-16079@sha256:9d2bf610e530f47c7e43df9719717075e4c7ec510aea6047b29208bffe7f1978 exists in daemon, skipping load
I0330 09:29:18.260674 42664 cache.go:193] Successfully downloaded all kic artifacts
I0330 09:29:18.260717 42664 start.go:364] acquiring machines lock for old-k8s-version-331000: {Name:mk68a72133bfb0ba0e52354dae23a3d4710ac349 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0330 09:29:18.260870 42664 start.go:368] acquired machines lock for "old-k8s-version-331000" in 141.417µs
I0330 09:29:18.260900 42664 start.go:93] Provisioning new machine with config: &{Name:old-k8s-version-331000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.37-1679075007-16079@sha256:9d2bf610e530f47c7e43df9719717075e4c7ec510aea6047b29208bffe7f1978 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-331000 Namespace:default APIServer
Name:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:f
alse DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP:} &{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}
I0330 09:29:18.260991 42664 start.go:125] createHost starting for "" (driver="docker")
I0330 09:29:18.282639 42664 out.go:204] * Creating docker container (CPUs=2, Memory=2200MB) ...
I0330 09:29:18.282836 42664 start.go:159] libmachine.API.Create for "old-k8s-version-331000" (driver="docker")
I0330 09:29:18.282864 42664 client.go:168] LocalClient.Create starting
I0330 09:29:18.282945 42664 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/16199-24978/.minikube/certs/ca.pem
I0330 09:29:18.282979 42664 main.go:141] libmachine: Decoding PEM data...
I0330 09:29:18.282996 42664 main.go:141] libmachine: Parsing certificate...
I0330 09:29:18.283063 42664 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/16199-24978/.minikube/certs/cert.pem
I0330 09:29:18.283087 42664 main.go:141] libmachine: Decoding PEM data...
I0330 09:29:18.283095 42664 main.go:141] libmachine: Parsing certificate...
I0330 09:29:18.283605 42664 cli_runner.go:164] Run: docker network inspect old-k8s-version-331000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
W0330 09:29:18.348067 42664 cli_runner.go:211] docker network inspect old-k8s-version-331000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1
I0330 09:29:18.348168 42664 network_create.go:281] running [docker network inspect old-k8s-version-331000] to gather additional debugging logs...
I0330 09:29:18.348184 42664 cli_runner.go:164] Run: docker network inspect old-k8s-version-331000
W0330 09:29:18.408334 42664 cli_runner.go:211] docker network inspect old-k8s-version-331000 returned with exit code 1
I0330 09:29:18.408363 42664 network_create.go:284] error running [docker network inspect old-k8s-version-331000]: docker network inspect old-k8s-version-331000: exit status 1
stdout:
[]
stderr:
Error: No such network: old-k8s-version-331000
I0330 09:29:18.408377 42664 network_create.go:286] output of [docker network inspect old-k8s-version-331000]: -- stdout --
[]
-- /stdout --
** stderr **
Error: No such network: old-k8s-version-331000
** /stderr **
I0330 09:29:18.408460 42664 cli_runner.go:164] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0330 09:29:18.471769 42664 network.go:212] skipping subnet 192.168.49.0/24 that is reserved: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0330 09:29:18.473349 42664 network.go:212] skipping subnet 192.168.58.0/24 that is reserved: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0330 09:29:18.474879 42664 network.go:212] skipping subnet 192.168.67.0/24 that is reserved: &{IP:192.168.67.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.67.0/24 Gateway:192.168.67.1 ClientMin:192.168.67.2 ClientMax:192.168.67.254 Broadcast:192.168.67.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0330 09:29:18.475189 42664 network.go:209] using free private subnet 192.168.76.0/24: &{IP:192.168.76.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.76.0/24 Gateway:192.168.76.1 ClientMin:192.168.76.2 ClientMax:192.168.76.254 Broadcast:192.168.76.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc000da97c0}
I0330 09:29:18.475203 42664 network_create.go:123] attempt to create docker network old-k8s-version-331000 192.168.76.0/24 with gateway 192.168.76.1 and MTU of 1500 ...
I0330 09:29:18.475275 42664 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.76.0/24 --gateway=192.168.76.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=1500 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-331000 old-k8s-version-331000
I0330 09:29:18.575264 42664 network_create.go:107] docker network old-k8s-version-331000 192.168.76.0/24 created
I0330 09:29:18.575309 42664 kic.go:117] calculated static IP "192.168.76.2" for the "old-k8s-version-331000" container
I0330 09:29:18.575427 42664 cli_runner.go:164] Run: docker ps -a --format {{.Names}}
I0330 09:29:18.637501 42664 cli_runner.go:164] Run: docker volume create old-k8s-version-331000 --label name.minikube.sigs.k8s.io=old-k8s-version-331000 --label created_by.minikube.sigs.k8s.io=true
I0330 09:29:18.707749 42664 oci.go:103] Successfully created a docker volume old-k8s-version-331000
I0330 09:29:18.707859 42664 cli_runner.go:164] Run: docker run --rm --name old-k8s-version-331000-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-331000 --entrypoint /usr/bin/test -v old-k8s-version-331000:/var gcr.io/k8s-minikube/kicbase-builds:v0.0.37-1679075007-16079@sha256:9d2bf610e530f47c7e43df9719717075e4c7ec510aea6047b29208bffe7f1978 -d /var/lib
I0330 09:29:19.213875 42664 oci.go:107] Successfully prepared a docker volume old-k8s-version-331000
I0330 09:29:19.213921 42664 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0330 09:29:19.213937 42664 kic.go:190] Starting extracting preloaded images to volume ...
I0330 09:29:19.214064 42664 cli_runner.go:164] Run: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/16199-24978/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-331000:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.37-1679075007-16079@sha256:9d2bf610e530f47c7e43df9719717075e4c7ec510aea6047b29208bffe7f1978 -I lz4 -xf /preloaded.tar -C /extractDir
I0330 09:29:25.534782 42664 cli_runner.go:217] Completed: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/16199-24978/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-331000:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.37-1679075007-16079@sha256:9d2bf610e530f47c7e43df9719717075e4c7ec510aea6047b29208bffe7f1978 -I lz4 -xf /preloaded.tar -C /extractDir: (6.320515306s)
I0330 09:29:25.534809 42664 kic.go:199] duration metric: took 6.320822 seconds to extract preloaded images to volume
I0330 09:29:25.534915 42664 cli_runner.go:164] Run: docker info --format "'{{json .SecurityOptions}}'"
I0330 09:29:25.741417 42664 cli_runner.go:164] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname old-k8s-version-331000 --name old-k8s-version-331000 --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-331000 --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=old-k8s-version-331000 --network old-k8s-version-331000 --ip 192.168.76.2 --volume old-k8s-version-331000:/var --security-opt apparmor=unconfined --memory=2200mb --memory-swap=2200mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.37-1679075007-16079@sha256:9d2bf610e530f47c7e43df9719717075e4c7ec510aea6047b29208bffe7f1978
I0330 09:29:26.177842 42664 cli_runner.go:164] Run: docker container inspect old-k8s-version-331000 --format={{.State.Running}}
I0330 09:29:26.255508 42664 cli_runner.go:164] Run: docker container inspect old-k8s-version-331000 --format={{.State.Status}}
I0330 09:29:26.330188 42664 cli_runner.go:164] Run: docker exec old-k8s-version-331000 stat /var/lib/dpkg/alternatives/iptables
I0330 09:29:26.452806 42664 oci.go:144] the created container "old-k8s-version-331000" has a running status.
I0330 09:29:26.452838 42664 kic.go:221] Creating ssh key for kic: /Users/jenkins/minikube-integration/16199-24978/.minikube/machines/old-k8s-version-331000/id_rsa...
I0330 09:29:26.717498 42664 kic_runner.go:191] docker (temp): /Users/jenkins/minikube-integration/16199-24978/.minikube/machines/old-k8s-version-331000/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0330 09:29:26.827722 42664 cli_runner.go:164] Run: docker container inspect old-k8s-version-331000 --format={{.State.Status}}
I0330 09:29:26.891585 42664 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0330 09:29:26.891605 42664 kic_runner.go:114] Args: [docker exec --privileged old-k8s-version-331000 chown docker:docker /home/docker/.ssh/authorized_keys]
I0330 09:29:27.005891 42664 cli_runner.go:164] Run: docker container inspect old-k8s-version-331000 --format={{.State.Status}}
I0330 09:29:27.067502 42664 machine.go:88] provisioning docker machine ...
I0330 09:29:27.067550 42664 ubuntu.go:169] provisioning hostname "old-k8s-version-331000"
I0330 09:29:27.067665 42664 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-331000
I0330 09:29:27.135603 42664 main.go:141] libmachine: Using SSH client type: native
I0330 09:29:27.135991 42664 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x140cc00] 0x140fca0 <nil> [] 0s} 127.0.0.1 58897 <nil> <nil>}
I0330 09:29:27.136014 42664 main.go:141] libmachine: About to run SSH command:
sudo hostname old-k8s-version-331000 && echo "old-k8s-version-331000" | sudo tee /etc/hostname
I0330 09:29:27.263548 42664 main.go:141] libmachine: SSH cmd err, output: <nil>: old-k8s-version-331000
I0330 09:29:27.284284 42664 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-331000
I0330 09:29:27.348379 42664 main.go:141] libmachine: Using SSH client type: native
I0330 09:29:27.348726 42664 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x140cc00] 0x140fca0 <nil> [] 0s} 127.0.0.1 58897 <nil> <nil>}
I0330 09:29:27.348745 42664 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sold-k8s-version-331000' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 old-k8s-version-331000/g' /etc/hosts;
else
echo '127.0.1.1 old-k8s-version-331000' | sudo tee -a /etc/hosts;
fi
fi
I0330 09:29:27.467571 42664 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0330 09:29:27.467593 42664 ubuntu.go:175] set auth options {CertDir:/Users/jenkins/minikube-integration/16199-24978/.minikube CaCertPath:/Users/jenkins/minikube-integration/16199-24978/.minikube/certs/ca.pem CaPrivateKeyPath:/Users/jenkins/minikube-integration/16199-24978/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/Users/jenkins/minikube-integration/16199-24978/.minikube/machines/server.pem ServerKeyPath:/Users/jenkins/minikube-integration/16199-24978/.minikube/machines/server-key.pem ClientKeyPath:/Users/jenkins/minikube-integration/16199-24978/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/Users/jenkins/minikube-integration/16199-24978/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/Users/jenkins/minikube-integration/16199-24978/.minikube}
I0330 09:29:27.467620 42664 ubuntu.go:177] setting up certificates
I0330 09:29:27.467631 42664 provision.go:83] configureAuth start
I0330 09:29:27.467716 42664 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-331000
I0330 09:29:27.533005 42664 provision.go:138] copyHostCerts
I0330 09:29:27.533109 42664 exec_runner.go:144] found /Users/jenkins/minikube-integration/16199-24978/.minikube/ca.pem, removing ...
I0330 09:29:27.533117 42664 exec_runner.go:207] rm: /Users/jenkins/minikube-integration/16199-24978/.minikube/ca.pem
I0330 09:29:27.533218 42664 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/16199-24978/.minikube/certs/ca.pem --> /Users/jenkins/minikube-integration/16199-24978/.minikube/ca.pem (1078 bytes)
I0330 09:29:27.533419 42664 exec_runner.go:144] found /Users/jenkins/minikube-integration/16199-24978/.minikube/cert.pem, removing ...
I0330 09:29:27.533425 42664 exec_runner.go:207] rm: /Users/jenkins/minikube-integration/16199-24978/.minikube/cert.pem
I0330 09:29:27.533490 42664 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/16199-24978/.minikube/certs/cert.pem --> /Users/jenkins/minikube-integration/16199-24978/.minikube/cert.pem (1123 bytes)
I0330 09:29:27.533649 42664 exec_runner.go:144] found /Users/jenkins/minikube-integration/16199-24978/.minikube/key.pem, removing ...
I0330 09:29:27.533655 42664 exec_runner.go:207] rm: /Users/jenkins/minikube-integration/16199-24978/.minikube/key.pem
I0330 09:29:27.533716 42664 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/16199-24978/.minikube/certs/key.pem --> /Users/jenkins/minikube-integration/16199-24978/.minikube/key.pem (1675 bytes)
I0330 09:29:27.533837 42664 provision.go:112] generating server cert: /Users/jenkins/minikube-integration/16199-24978/.minikube/machines/server.pem ca-key=/Users/jenkins/minikube-integration/16199-24978/.minikube/certs/ca.pem private-key=/Users/jenkins/minikube-integration/16199-24978/.minikube/certs/ca-key.pem org=jenkins.old-k8s-version-331000 san=[192.168.76.2 127.0.0.1 localhost 127.0.0.1 minikube old-k8s-version-331000]
I0330 09:29:27.729549 42664 provision.go:172] copyRemoteCerts
I0330 09:29:27.729614 42664 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0330 09:29:27.729680 42664 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-331000
I0330 09:29:27.795656 42664 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:58897 SSHKeyPath:/Users/jenkins/minikube-integration/16199-24978/.minikube/machines/old-k8s-version-331000/id_rsa Username:docker}
I0330 09:29:27.885221 42664 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16199-24978/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1078 bytes)
I0330 09:29:27.903795 42664 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16199-24978/.minikube/machines/server.pem --> /etc/docker/server.pem (1241 bytes)
I0330 09:29:27.921549 42664 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16199-24978/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1679 bytes)
I0330 09:29:27.939420 42664 provision.go:86] duration metric: configureAuth took 471.759078ms
I0330 09:29:27.939435 42664 ubuntu.go:193] setting minikube options for container-runtime
I0330 09:29:27.939587 42664 config.go:182] Loaded profile config "old-k8s-version-331000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.16.0
I0330 09:29:27.939659 42664 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-331000
I0330 09:29:28.008366 42664 main.go:141] libmachine: Using SSH client type: native
I0330 09:29:28.008896 42664 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x140cc00] 0x140fca0 <nil> [] 0s} 127.0.0.1 58897 <nil> <nil>}
I0330 09:29:28.008912 42664 main.go:141] libmachine: About to run SSH command:
df --output=fstype / | tail -n 1
I0330 09:29:28.125181 42664 main.go:141] libmachine: SSH cmd err, output: <nil>: overlay
I0330 09:29:28.125199 42664 ubuntu.go:71] root file system type: overlay
I0330 09:29:28.125320 42664 provision.go:309] Updating docker unit: /lib/systemd/system/docker.service ...
I0330 09:29:28.125426 42664 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-331000
I0330 09:29:28.188947 42664 main.go:141] libmachine: Using SSH client type: native
I0330 09:29:28.189345 42664 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x140cc00] 0x140fca0 <nil> [] 0s} 127.0.0.1 58897 <nil> <nil>}
I0330 09:29:28.189403 42664 main.go:141] libmachine: About to run SSH command:
sudo mkdir -p /lib/systemd/system && printf %s "[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP \$MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service.new
I0330 09:29:28.316210 42664 main.go:141] libmachine: SSH cmd err, output: <nil>: [Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
I0330 09:29:28.316311 42664 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-331000
I0330 09:29:28.377705 42664 main.go:141] libmachine: Using SSH client type: native
I0330 09:29:28.378068 42664 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x140cc00] 0x140fca0 <nil> [] 0s} 127.0.0.1 58897 <nil> <nil>}
I0330 09:29:28.378081 42664 main.go:141] libmachine: About to run SSH command:
sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
I0330 09:29:29.009951 42664 main.go:141] libmachine: SSH cmd err, output: <nil>: --- /lib/systemd/system/docker.service 2023-02-09 19:46:56.000000000 +0000
+++ /lib/systemd/system/docker.service.new 2023-03-30 16:29:28.313246731 +0000
@@ -1,30 +1,32 @@
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
-After=network-online.target docker.socket firewalld.service containerd.service time-set.target
-Wants=network-online.target containerd.service
+BindsTo=containerd.service
+After=network-online.target firewalld.service containerd.service
+Wants=network-online.target
Requires=docker.socket
+StartLimitBurst=3
+StartLimitIntervalSec=60
[Service]
Type=notify
-# the default is not to use systemd for cgroups because the delegate issues still
-# exists and systemd currently does not support the cgroup feature set required
-# for containers run by docker
-ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
-ExecReload=/bin/kill -s HUP $MAINPID
-TimeoutStartSec=0
-RestartSec=2
-Restart=always
+Restart=on-failure
-# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
-# Both the old, and new location are accepted by systemd 229 and up, so using the old location
-# to make them work for either version of systemd.
-StartLimitBurst=3
-# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
-# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
-# this option work for either version of systemd.
-StartLimitInterval=60s
+
+# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
+# The base configuration already specifies an 'ExecStart=...' command. The first directive
+# here is to clear out that command inherited from the base configuration. Without this,
+# the command from the base configuration and the command specified here are treated as
+# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
+# will catch this invalid input and refuse to start the service with an error like:
+# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
+
+# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
+# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
+ExecStart=
+ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
+ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
@@ -32,16 +34,16 @@
LimitNPROC=infinity
LimitCORE=infinity
-# Comment TasksMax if your systemd version does not support it.
-# Only systemd 226 and above support this option.
+# Uncomment TasksMax if your systemd version supports it.
+# Only systemd 226 and above support this version.
TasksMax=infinity
+TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
-OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
I0330 09:29:29.009971 42664 machine.go:91] provisioned docker machine in 1.94242981s
I0330 09:29:29.009980 42664 client.go:171] LocalClient.Create took 10.727031116s
I0330 09:29:29.009998 42664 start.go:167] duration metric: libmachine.API.Create for "old-k8s-version-331000" took 10.727084257s
I0330 09:29:29.010009 42664 start.go:300] post-start starting for "old-k8s-version-331000" (driver="docker")
I0330 09:29:29.010019 42664 start.go:328] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0330 09:29:29.010102 42664 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0330 09:29:29.010163 42664 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-331000
I0330 09:29:29.084773 42664 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:58897 SSHKeyPath:/Users/jenkins/minikube-integration/16199-24978/.minikube/machines/old-k8s-version-331000/id_rsa Username:docker}
I0330 09:29:29.174701 42664 ssh_runner.go:195] Run: cat /etc/os-release
I0330 09:29:29.178552 42664 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0330 09:29:29.178569 42664 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0330 09:29:29.178577 42664 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0330 09:29:29.178582 42664 info.go:137] Remote host: Ubuntu 20.04.5 LTS
I0330 09:29:29.178592 42664 filesync.go:126] Scanning /Users/jenkins/minikube-integration/16199-24978/.minikube/addons for local assets ...
I0330 09:29:29.178683 42664 filesync.go:126] Scanning /Users/jenkins/minikube-integration/16199-24978/.minikube/files for local assets ...
I0330 09:29:29.178857 42664 filesync.go:149] local asset: /Users/jenkins/minikube-integration/16199-24978/.minikube/files/etc/ssl/certs/254482.pem -> 254482.pem in /etc/ssl/certs
I0330 09:29:29.179043 42664 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0330 09:29:29.187821 42664 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16199-24978/.minikube/files/etc/ssl/certs/254482.pem --> /etc/ssl/certs/254482.pem (1708 bytes)
I0330 09:29:29.208587 42664 start.go:303] post-start completed in 198.567296ms
I0330 09:29:29.209152 42664 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-331000
I0330 09:29:29.281261 42664 profile.go:148] Saving config to /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/config.json ...
I0330 09:29:29.281786 42664 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0330 09:29:29.281871 42664 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-331000
I0330 09:29:29.346709 42664 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:58897 SSHKeyPath:/Users/jenkins/minikube-integration/16199-24978/.minikube/machines/old-k8s-version-331000/id_rsa Username:docker}
I0330 09:29:29.434477 42664 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0330 09:29:29.439803 42664 start.go:128] duration metric: createHost completed in 11.178717519s
I0330 09:29:29.439826 42664 start.go:83] releasing machines lock for "old-k8s-version-331000", held for 11.178864098s
I0330 09:29:29.439933 42664 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-331000
I0330 09:29:29.511115 42664 ssh_runner.go:195] Run: cat /version.json
I0330 09:29:29.511134 42664 ssh_runner.go:195] Run: curl -sS -m 2 https://k8s.gcr.io/
I0330 09:29:29.511182 42664 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-331000
I0330 09:29:29.511232 42664 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-331000
I0330 09:29:29.587480 42664 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:58897 SSHKeyPath:/Users/jenkins/minikube-integration/16199-24978/.minikube/machines/old-k8s-version-331000/id_rsa Username:docker}
I0330 09:29:29.587560 42664 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:58897 SSHKeyPath:/Users/jenkins/minikube-integration/16199-24978/.minikube/machines/old-k8s-version-331000/id_rsa Username:docker}
I0330 09:29:29.673232 42664 ssh_runner.go:195] Run: systemctl --version
I0330 09:29:29.953982 42664 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
I0330 09:29:29.959544 42664 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *loopback.conf* -not -name *.mk_disabled -exec sh -c "grep -q loopback {} && ( grep -q name {} || sudo sed -i '/"type": "loopback"/i \ \ \ \ "name": "loopback",' {} ) && sudo sed -i 's|"cniVersion": ".*"|"cniVersion": "1.0.0"|g' {}" ;
I0330 09:29:29.980911 42664 cni.go:229] loopback cni configuration patched: "/etc/cni/net.d/*loopback.conf*" found
I0330 09:29:29.980982 42664 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *bridge* -not -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e '/"dst": ".*:.*"/d' -e 's|^(.*)"dst": (.*)[,*]$|\1"dst": \2|g' -e '/"subnet": ".*:.*"/d' -e 's|^(.*)"subnet": ".*"(.*)[,*]$|\1"subnet": "10.244.0.0/16"\2|g' {}" ;
I0330 09:29:29.995708 42664 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e 's|^(.*)"subnet": ".*"(.*)$|\1"subnet": "10.244.0.0/16"\2|g' -e 's|^(.*)"gateway": ".*"(.*)$|\1"gateway": "10.244.0.1"\2|g' {}" ;
I0330 09:29:30.004022 42664 cni.go:307] configured [/etc/cni/net.d/100-crio-bridge.conf] bridge cni config(s)
I0330 09:29:30.004035 42664 start.go:481] detecting cgroup driver to use...
I0330 09:29:30.004045 42664 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0330 09:29:30.004123 42664 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0330 09:29:30.018106 42664 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "k8s.gcr.io/pause:3.1"|' /etc/containerd/config.toml"
I0330 09:29:30.026834 42664 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I0330 09:29:30.036205 42664 containerd.go:145] configuring containerd to use "cgroupfs" as cgroup driver...
I0330 09:29:30.036263 42664 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = false|g' /etc/containerd/config.toml"
I0330 09:29:30.045742 42664 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0330 09:29:30.054898 42664 ssh_runner.go:195] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I0330 09:29:30.063787 42664 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0330 09:29:30.072921 42664 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0330 09:29:30.081036 42664 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I0330 09:29:30.089998 42664 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0330 09:29:30.097765 42664 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0330 09:29:30.105245 42664 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0330 09:29:30.183045 42664 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0330 09:29:30.258763 42664 start.go:481] detecting cgroup driver to use...
I0330 09:29:30.258784 42664 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0330 09:29:30.258853 42664 ssh_runner.go:195] Run: sudo systemctl cat docker.service
I0330 09:29:30.275377 42664 cruntime.go:276] skipping containerd shutdown because we are bound to it
I0330 09:29:30.275450 42664 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0330 09:29:30.285703 42664 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///var/run/dockershim.sock
" | sudo tee /etc/crictl.yaml"
I0330 09:29:30.303406 42664 ssh_runner.go:195] Run: which cri-dockerd
I0330 09:29:30.307925 42664 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/cri-docker.service.d
I0330 09:29:30.316915 42664 ssh_runner.go:362] scp memory --> /etc/systemd/system/cri-docker.service.d/10-cni.conf (184 bytes)
I0330 09:29:30.346978 42664 ssh_runner.go:195] Run: sudo systemctl unmask docker.service
I0330 09:29:30.455524 42664 ssh_runner.go:195] Run: sudo systemctl enable docker.socket
I0330 09:29:30.519704 42664 docker.go:538] configuring docker to use "cgroupfs" as cgroup driver...
I0330 09:29:30.519728 42664 ssh_runner.go:362] scp memory --> /etc/docker/daemon.json (144 bytes)
I0330 09:29:30.549528 42664 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0330 09:29:30.640939 42664 ssh_runner.go:195] Run: sudo systemctl restart docker
I0330 09:29:30.872225 42664 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0330 09:29:30.900191 42664 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0330 09:29:30.972076 42664 out.go:204] * Preparing Kubernetes v1.16.0 on Docker 23.0.1 ...
I0330 09:29:30.972195 42664 cli_runner.go:164] Run: docker exec -t old-k8s-version-331000 dig +short host.docker.internal
I0330 09:29:31.092699 42664 network.go:96] got host ip for mount in container by digging dns: 192.168.65.2
I0330 09:29:31.092822 42664 ssh_runner.go:195] Run: grep 192.168.65.2 host.minikube.internal$ /etc/hosts
I0330 09:29:31.097414 42664 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.65.2 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0330 09:29:31.107597 42664 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "8443/tcp") 0).HostPort}}'" old-k8s-version-331000
I0330 09:29:31.168457 42664 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0330 09:29:31.168539 42664 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0330 09:29:31.189550 42664 docker.go:639] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0330 09:29:31.189574 42664 docker.go:569] Images already preloaded, skipping extraction
I0330 09:29:31.189680 42664 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0330 09:29:31.210974 42664 docker.go:639] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0330 09:29:31.210993 42664 cache_images.go:84] Images are preloaded, skipping loading
I0330 09:29:31.211108 42664 ssh_runner.go:195] Run: docker info --format {{.CgroupDriver}}
I0330 09:29:31.237611 42664 cni.go:84] Creating CNI manager for ""
I0330 09:29:31.237629 42664 cni.go:161] CNI unnecessary in this configuration, recommending no CNI
I0330 09:29:31.237647 42664 kubeadm.go:87] Using pod CIDR: 10.244.0.0/16
I0330 09:29:31.237666 42664 kubeadm.go:172] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.76.2 APIServerPort:8443 KubernetesVersion:v1.16.0 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:old-k8s-version-331000 NodeName:old-k8s-version-331000 DNSDomain:cluster.local CRISocket:/var/run/dockershim.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.76.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.76.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticP
odPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[hairpinMode:hairpin-veth runtimeRequestTimeout:15m]}
I0330 09:29:31.237791 42664 kubeadm.go:177] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.76.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: "old-k8s-version-331000"
kubeletExtraArgs:
node-ip: 192.168.76.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.76.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: old-k8s-version-331000
controlPlaneEndpoint: control-plane.minikube.internal:8443
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
listen-metrics-urls: http://127.0.0.1:2381,http://192.168.76.2:2381
kubernetesVersion: v1.16.0
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0330 09:29:31.237881 42664 kubeadm.go:968] kubelet [Unit]
Wants=docker.socket
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.16.0/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=old-k8s-version-331000 --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.76.2
[Install]
config:
{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-331000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:}
I0330 09:29:31.237944 42664 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.16.0
I0330 09:29:31.246309 42664 binaries.go:44] Found k8s binaries, skipping transfer
I0330 09:29:31.246373 42664 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0330 09:29:31.254286 42664 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (348 bytes)
I0330 09:29:31.267777 42664 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0330 09:29:31.280945 42664 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2174 bytes)
I0330 09:29:31.294259 42664 ssh_runner.go:195] Run: grep 192.168.76.2 control-plane.minikube.internal$ /etc/hosts
I0330 09:29:31.298282 42664 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.76.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0330 09:29:31.308439 42664 certs.go:56] Setting up /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000 for IP: 192.168.76.2
I0330 09:29:31.308457 42664 certs.go:186] acquiring lock for shared ca certs: {Name:mk7bf1a10342abaa451a7b833b2ba3f85c81aeda Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0330 09:29:31.308620 42664 certs.go:195] skipping minikubeCA CA generation: /Users/jenkins/minikube-integration/16199-24978/.minikube/ca.key
I0330 09:29:31.308671 42664 certs.go:195] skipping proxyClientCA CA generation: /Users/jenkins/minikube-integration/16199-24978/.minikube/proxy-client-ca.key
I0330 09:29:31.308711 42664 certs.go:315] generating minikube-user signed cert: /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/client.key
I0330 09:29:31.308727 42664 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/client.crt with IP's: []
I0330 09:29:31.428414 42664 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/client.crt ...
I0330 09:29:31.428426 42664 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/client.crt: {Name:mk65e4cbf4477d3526bca1450688ad0b6947726f Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0330 09:29:31.428734 42664 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/client.key ...
I0330 09:29:31.428745 42664 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/client.key: {Name:mkbd3cf09b0e2df71db6a082f0c5a57bc353d6a3 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0330 09:29:31.428945 42664 certs.go:315] generating minikube signed cert: /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/apiserver.key.31bdca25
I0330 09:29:31.428960 42664 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/apiserver.crt.31bdca25 with IP's: [192.168.76.2 10.96.0.1 127.0.0.1 10.0.0.1]
I0330 09:29:31.868415 42664 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/apiserver.crt.31bdca25 ...
I0330 09:29:31.868437 42664 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/apiserver.crt.31bdca25: {Name:mk3733d0fa932234fe167beba04ea58bd7a5231f Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0330 09:29:31.868787 42664 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/apiserver.key.31bdca25 ...
I0330 09:29:31.868799 42664 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/apiserver.key.31bdca25: {Name:mkb723a7880d6710c7ee3ee38946a9ff822b5ea9 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0330 09:29:31.869012 42664 certs.go:333] copying /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/apiserver.crt.31bdca25 -> /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/apiserver.crt
I0330 09:29:31.869185 42664 certs.go:337] copying /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/apiserver.key.31bdca25 -> /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/apiserver.key
I0330 09:29:31.869356 42664 certs.go:315] generating aggregator signed cert: /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/proxy-client.key
I0330 09:29:31.869373 42664 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/proxy-client.crt with IP's: []
I0330 09:29:31.998299 42664 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/proxy-client.crt ...
I0330 09:29:31.998313 42664 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/proxy-client.crt: {Name:mk88414fe7fc2d889797f3827c158d24911f0fa6 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0330 09:29:31.998609 42664 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/proxy-client.key ...
I0330 09:29:31.998622 42664 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/proxy-client.key: {Name:mkb624882b987be5bcbaac72ff2957d3c90ba452 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0330 09:29:31.999035 42664 certs.go:401] found cert: /Users/jenkins/minikube-integration/16199-24978/.minikube/certs/Users/jenkins/minikube-integration/16199-24978/.minikube/certs/25448.pem (1338 bytes)
W0330 09:29:31.999082 42664 certs.go:397] ignoring /Users/jenkins/minikube-integration/16199-24978/.minikube/certs/Users/jenkins/minikube-integration/16199-24978/.minikube/certs/25448_empty.pem, impossibly tiny 0 bytes
I0330 09:29:31.999099 42664 certs.go:401] found cert: /Users/jenkins/minikube-integration/16199-24978/.minikube/certs/Users/jenkins/minikube-integration/16199-24978/.minikube/certs/ca-key.pem (1679 bytes)
I0330 09:29:31.999133 42664 certs.go:401] found cert: /Users/jenkins/minikube-integration/16199-24978/.minikube/certs/Users/jenkins/minikube-integration/16199-24978/.minikube/certs/ca.pem (1078 bytes)
I0330 09:29:31.999168 42664 certs.go:401] found cert: /Users/jenkins/minikube-integration/16199-24978/.minikube/certs/Users/jenkins/minikube-integration/16199-24978/.minikube/certs/cert.pem (1123 bytes)
I0330 09:29:31.999199 42664 certs.go:401] found cert: /Users/jenkins/minikube-integration/16199-24978/.minikube/certs/Users/jenkins/minikube-integration/16199-24978/.minikube/certs/key.pem (1675 bytes)
I0330 09:29:31.999269 42664 certs.go:401] found cert: /Users/jenkins/minikube-integration/16199-24978/.minikube/files/etc/ssl/certs/Users/jenkins/minikube-integration/16199-24978/.minikube/files/etc/ssl/certs/254482.pem (1708 bytes)
I0330 09:29:31.999773 42664 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1399 bytes)
I0330 09:29:32.019348 42664 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1679 bytes)
I0330 09:29:32.037327 42664 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0330 09:29:32.055165 42664 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16199-24978/.minikube/profiles/old-k8s-version-331000/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1679 bytes)
I0330 09:29:32.073304 42664 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16199-24978/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0330 09:29:32.091521 42664 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16199-24978/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1679 bytes)
I0330 09:29:32.109521 42664 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16199-24978/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0330 09:29:32.127152 42664 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16199-24978/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1675 bytes)
I0330 09:29:32.145062 42664 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16199-24978/.minikube/files/etc/ssl/certs/254482.pem --> /usr/share/ca-certificates/254482.pem (1708 bytes)
I0330 09:29:32.162990 42664 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16199-24978/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0330 09:29:32.181813 42664 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16199-24978/.minikube/certs/25448.pem --> /usr/share/ca-certificates/25448.pem (1338 bytes)
I0330 09:29:32.199569 42664 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0330 09:29:32.212998 42664 ssh_runner.go:195] Run: openssl version
I0330 09:29:32.218877 42664 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0330 09:29:32.227196 42664 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0330 09:29:32.231166 42664 certs.go:444] hashing: -rw-r--r-- 1 root root 1111 Mar 30 15:39 /usr/share/ca-certificates/minikubeCA.pem
I0330 09:29:32.231212 42664 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0330 09:29:32.236897 42664 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0330 09:29:32.245377 42664 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/25448.pem && ln -fs /usr/share/ca-certificates/25448.pem /etc/ssl/certs/25448.pem"
I0330 09:29:32.253861 42664 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/25448.pem
I0330 09:29:32.272354 42664 certs.go:444] hashing: -rw-r--r-- 1 root root 1338 Mar 30 15:43 /usr/share/ca-certificates/25448.pem
I0330 09:29:32.272413 42664 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/25448.pem
I0330 09:29:32.278115 42664 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/25448.pem /etc/ssl/certs/51391683.0"
I0330 09:29:32.286438 42664 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/254482.pem && ln -fs /usr/share/ca-certificates/254482.pem /etc/ssl/certs/254482.pem"
I0330 09:29:32.295036 42664 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/254482.pem
I0330 09:29:32.299195 42664 certs.go:444] hashing: -rw-r--r-- 1 root root 1708 Mar 30 15:43 /usr/share/ca-certificates/254482.pem
I0330 09:29:32.299239 42664 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/254482.pem
I0330 09:29:32.304754 42664 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/254482.pem /etc/ssl/certs/3ec20f2e.0"
I0330 09:29:32.313092 42664 kubeadm.go:401] StartCluster: {Name:old-k8s-version-331000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.37-1679075007-16079@sha256:9d2bf610e530f47c7e43df9719717075e4c7ec510aea6047b29208bffe7f1978 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-331000 Namespace:default APIServerName:minikubeCA APISer
verNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false Disab
leMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP:}
I0330 09:29:32.313193 42664 ssh_runner.go:195] Run: docker ps --filter status=paused --filter=name=k8s_.*_(kube-system)_ --format={{.ID}}
I0330 09:29:32.332562 42664 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0330 09:29:32.341052 42664 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0330 09:29:32.349564 42664 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0330 09:29:32.349623 42664 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0330 09:29:32.358050 42664 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0330 09:29:32.358080 42664 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0330 09:29:32.413758 42664 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0330 09:29:32.413798 42664 kubeadm.go:322] [preflight] Running pre-flight checks
I0330 09:29:32.587773 42664 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0330 09:29:32.587867 42664 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0330 09:29:32.587942 42664 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0330 09:29:32.747332 42664 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0330 09:29:32.748122 42664 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0330 09:29:32.754534 42664 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0330 09:29:32.828485 42664 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0330 09:29:32.850163 42664 out.go:204] - Generating certificates and keys ...
I0330 09:29:32.850256 42664 kubeadm.go:322] [certs] Using existing ca certificate authority
I0330 09:29:32.850333 42664 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0330 09:29:32.968995 42664 kubeadm.go:322] [certs] Generating "apiserver-kubelet-client" certificate and key
I0330 09:29:33.150535 42664 kubeadm.go:322] [certs] Generating "front-proxy-ca" certificate and key
I0330 09:29:33.442734 42664 kubeadm.go:322] [certs] Generating "front-proxy-client" certificate and key
I0330 09:29:33.736156 42664 kubeadm.go:322] [certs] Generating "etcd/ca" certificate and key
I0330 09:29:33.938772 42664 kubeadm.go:322] [certs] Generating "etcd/server" certificate and key
I0330 09:29:33.938881 42664 kubeadm.go:322] [certs] etcd/server serving cert is signed for DNS names [old-k8s-version-331000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
I0330 09:29:34.066403 42664 kubeadm.go:322] [certs] Generating "etcd/peer" certificate and key
I0330 09:29:34.066529 42664 kubeadm.go:322] [certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-331000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
I0330 09:29:34.132383 42664 kubeadm.go:322] [certs] Generating "etcd/healthcheck-client" certificate and key
I0330 09:29:34.427492 42664 kubeadm.go:322] [certs] Generating "apiserver-etcd-client" certificate and key
I0330 09:29:34.561151 42664 kubeadm.go:322] [certs] Generating "sa" key and public key
I0330 09:29:34.561206 42664 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0330 09:29:34.670189 42664 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0330 09:29:34.889689 42664 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0330 09:29:35.040075 42664 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0330 09:29:35.141125 42664 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0330 09:29:35.141836 42664 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0330 09:29:35.163269 42664 out.go:204] - Booting up control plane ...
I0330 09:29:35.163417 42664 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0330 09:29:35.163588 42664 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0330 09:29:35.163705 42664 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0330 09:29:35.163853 42664 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0330 09:29:35.164109 42664 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0330 09:30:15.150328 42664 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0330 09:30:15.150967 42664 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0330 09:30:15.151217 42664 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0330 09:30:20.153763 42664 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0330 09:30:20.153909 42664 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0330 09:30:30.154359 42664 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0330 09:30:30.154504 42664 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0330 09:30:50.155144 42664 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0330 09:30:50.155329 42664 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0330 09:31:30.156271 42664 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0330 09:31:30.156469 42664 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0330 09:31:30.156482 42664 kubeadm.go:322]
I0330 09:31:30.156513 42664 kubeadm.go:322] Unfortunately, an error has occurred:
I0330 09:31:30.156550 42664 kubeadm.go:322] timed out waiting for the condition
I0330 09:31:30.156557 42664 kubeadm.go:322]
I0330 09:31:30.156596 42664 kubeadm.go:322] This error is likely caused by:
I0330 09:31:30.156634 42664 kubeadm.go:322] - The kubelet is not running
I0330 09:31:30.156739 42664 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0330 09:31:30.156753 42664 kubeadm.go:322]
I0330 09:31:30.156831 42664 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0330 09:31:30.156854 42664 kubeadm.go:322] - 'systemctl status kubelet'
I0330 09:31:30.156878 42664 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0330 09:31:30.156882 42664 kubeadm.go:322]
I0330 09:31:30.156955 42664 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0330 09:31:30.157027 42664 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0330 09:31:30.157094 42664 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0330 09:31:30.157141 42664 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0330 09:31:30.157201 42664 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0330 09:31:30.157231 42664 kubeadm.go:322] - 'docker logs CONTAINERID'
I0330 09:31:30.160198 42664 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0330 09:31:30.160294 42664 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0330 09:31:30.160431 42664 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.1. Latest validated version: 18.09
I0330 09:31:30.160505 42664 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0330 09:31:30.160596 42664 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0330 09:31:30.160659 42664 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
W0330 09:31:30.160796 42664 out.go:239] ! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-331000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-331000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.1. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-331000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-331000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.1. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
I0330 09:31:30.160829 42664 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm reset --cri-socket /var/run/dockershim.sock --force"
I0330 09:31:30.574701 42664 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service kubelet
I0330 09:31:30.585699 42664 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0330 09:31:30.585756 42664 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0330 09:31:30.593914 42664 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0330 09:31:30.593937 42664 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0330 09:31:30.648249 42664 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0330 09:31:30.648312 42664 kubeadm.go:322] [preflight] Running pre-flight checks
I0330 09:31:30.827821 42664 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0330 09:31:30.827897 42664 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0330 09:31:30.827964 42664 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0330 09:31:30.991884 42664 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0330 09:31:30.992518 42664 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0330 09:31:30.999817 42664 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0330 09:31:31.074920 42664 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0330 09:31:31.100837 42664 out.go:204] - Generating certificates and keys ...
I0330 09:31:31.100954 42664 kubeadm.go:322] [certs] Using existing ca certificate authority
I0330 09:31:31.101051 42664 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0330 09:31:31.101132 42664 kubeadm.go:322] [certs] Using existing apiserver-kubelet-client certificate and key on disk
I0330 09:31:31.101184 42664 kubeadm.go:322] [certs] Using existing front-proxy-ca certificate authority
I0330 09:31:31.101260 42664 kubeadm.go:322] [certs] Using existing front-proxy-client certificate and key on disk
I0330 09:31:31.101350 42664 kubeadm.go:322] [certs] Using existing etcd/ca certificate authority
I0330 09:31:31.101411 42664 kubeadm.go:322] [certs] Using existing etcd/server certificate and key on disk
I0330 09:31:31.101455 42664 kubeadm.go:322] [certs] Using existing etcd/peer certificate and key on disk
I0330 09:31:31.101545 42664 kubeadm.go:322] [certs] Using existing etcd/healthcheck-client certificate and key on disk
I0330 09:31:31.101616 42664 kubeadm.go:322] [certs] Using existing apiserver-etcd-client certificate and key on disk
I0330 09:31:31.101646 42664 kubeadm.go:322] [certs] Using the existing "sa" key
I0330 09:31:31.101708 42664 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0330 09:31:31.176786 42664 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0330 09:31:31.263675 42664 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0330 09:31:31.385623 42664 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0330 09:31:31.599691 42664 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0330 09:31:31.600400 42664 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0330 09:31:31.621041 42664 out.go:204] - Booting up control plane ...
I0330 09:31:31.621157 42664 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0330 09:31:31.621247 42664 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0330 09:31:31.621328 42664 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0330 09:31:31.621418 42664 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0330 09:31:31.621598 42664 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0330 09:32:11.609707 42664 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0330 09:32:11.610700 42664 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0330 09:32:11.610929 42664 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0330 09:32:16.611587 42664 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0330 09:32:16.611739 42664 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0330 09:32:26.613189 42664 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0330 09:32:26.613345 42664 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0330 09:32:46.614002 42664 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0330 09:32:46.614195 42664 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0330 09:33:26.599903 42664 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0330 09:33:26.600129 42664 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0330 09:33:26.600149 42664 kubeadm.go:322]
I0330 09:33:26.600206 42664 kubeadm.go:322] Unfortunately, an error has occurred:
I0330 09:33:26.600306 42664 kubeadm.go:322] timed out waiting for the condition
I0330 09:33:26.600323 42664 kubeadm.go:322]
I0330 09:33:26.600368 42664 kubeadm.go:322] This error is likely caused by:
I0330 09:33:26.600409 42664 kubeadm.go:322] - The kubelet is not running
I0330 09:33:26.600585 42664 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0330 09:33:26.600658 42664 kubeadm.go:322]
I0330 09:33:26.600792 42664 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0330 09:33:26.600833 42664 kubeadm.go:322] - 'systemctl status kubelet'
I0330 09:33:26.600868 42664 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0330 09:33:26.600876 42664 kubeadm.go:322]
I0330 09:33:26.600984 42664 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0330 09:33:26.601087 42664 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0330 09:33:26.601206 42664 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0330 09:33:26.601248 42664 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0330 09:33:26.601326 42664 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0330 09:33:26.601357 42664 kubeadm.go:322] - 'docker logs CONTAINERID'
I0330 09:33:26.604051 42664 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0330 09:33:26.604125 42664 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0330 09:33:26.604249 42664 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.1. Latest validated version: 18.09
I0330 09:33:26.604329 42664 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0330 09:33:26.604405 42664 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0330 09:33:26.604463 42664 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
I0330 09:33:26.604484 42664 kubeadm.go:403] StartCluster complete in 3m54.306239996s
I0330 09:33:26.604590 42664 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-apiserver --format={{.ID}}
I0330 09:33:26.625195 42664 logs.go:277] 0 containers: []
W0330 09:33:26.625209 42664 logs.go:279] No container was found matching "kube-apiserver"
I0330 09:33:26.625277 42664 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_etcd --format={{.ID}}
I0330 09:33:26.646293 42664 logs.go:277] 0 containers: []
W0330 09:33:26.646306 42664 logs.go:279] No container was found matching "etcd"
I0330 09:33:26.646378 42664 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_coredns --format={{.ID}}
I0330 09:33:26.665884 42664 logs.go:277] 0 containers: []
W0330 09:33:26.665898 42664 logs.go:279] No container was found matching "coredns"
I0330 09:33:26.665965 42664 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-scheduler --format={{.ID}}
I0330 09:33:26.685549 42664 logs.go:277] 0 containers: []
W0330 09:33:26.685563 42664 logs.go:279] No container was found matching "kube-scheduler"
I0330 09:33:26.685636 42664 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-proxy --format={{.ID}}
I0330 09:33:26.706651 42664 logs.go:277] 0 containers: []
W0330 09:33:26.706663 42664 logs.go:279] No container was found matching "kube-proxy"
I0330 09:33:26.706729 42664 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-controller-manager --format={{.ID}}
I0330 09:33:26.726595 42664 logs.go:277] 0 containers: []
W0330 09:33:26.726608 42664 logs.go:279] No container was found matching "kube-controller-manager"
I0330 09:33:26.726680 42664 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kindnet --format={{.ID}}
I0330 09:33:26.747027 42664 logs.go:277] 0 containers: []
W0330 09:33:26.747041 42664 logs.go:279] No container was found matching "kindnet"
I0330 09:33:26.747048 42664 logs.go:123] Gathering logs for kubelet ...
I0330 09:33:26.747056 42664 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
I0330 09:33:26.787170 42664 logs.go:123] Gathering logs for dmesg ...
I0330 09:33:26.787186 42664 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0330 09:33:26.800626 42664 logs.go:123] Gathering logs for describe nodes ...
I0330 09:33:26.800640 42664 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0330 09:33:26.856478 42664 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0330 09:33:26.856489 42664 logs.go:123] Gathering logs for Docker ...
I0330 09:33:26.856497 42664 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u docker -u cri-docker -n 400"
I0330 09:33:26.880491 42664 logs.go:123] Gathering logs for container status ...
I0330 09:33:26.880505 42664 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0330 09:33:28.925736 42664 ssh_runner.go:235] Completed: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a": (2.04550699s)
W0330 09:33:28.925849 42664 out.go:369] Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.1. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0330 09:33:28.925863 42664 out.go:239] *
*
W0330 09:33:28.925970 42664 out.go:239] X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.1. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.1. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0330 09:33:28.925984 42664 out.go:239] *
*
W0330 09:33:28.926572 42664 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0330 09:33:29.005233 42664 out.go:177]
W0330 09:33:29.064426 42664 out.go:239] X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.1. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 23.0.1. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0330 09:33:29.064526 42664 out.go:239] * Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
* Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
W0330 09:33:29.064586 42664 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/4172
* Related issue: https://github.com/kubernetes/minikube/issues/4172
I0330 09:33:29.122403 42664 out.go:177]
** /stderr **
start_stop_delete_test.go:188: failed starting minikube -first start-. args "out/minikube-darwin-amd64 start -p old-k8s-version-331000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0": exit status 109
helpers_test.go:222: -----------------------post-mortem--------------------------------
helpers_test.go:230: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/FirstStart]: docker inspect <======
helpers_test.go:231: (dbg) Run: docker inspect old-k8s-version-331000
helpers_test.go:235: (dbg) docker inspect old-k8s-version-331000:
-- stdout --
[
{
"Id": "521ae416df386b804a8c63fde7d0580ea83db7526870fcff106c7ff7ac19c55a",
"Created": "2023-03-30T16:29:25.810047609Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 660363,
"ExitCode": 0,
"Error": "",
"StartedAt": "2023-03-30T16:29:26.166024196Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:9d2236b717ccec479afd77862e0eef2affb7c3e4fe7eecdc0546bff7b370db25",
"ResolvConfPath": "/var/lib/docker/containers/521ae416df386b804a8c63fde7d0580ea83db7526870fcff106c7ff7ac19c55a/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/521ae416df386b804a8c63fde7d0580ea83db7526870fcff106c7ff7ac19c55a/hostname",
"HostsPath": "/var/lib/docker/containers/521ae416df386b804a8c63fde7d0580ea83db7526870fcff106c7ff7ac19c55a/hosts",
"LogPath": "/var/lib/docker/containers/521ae416df386b804a8c63fde7d0580ea83db7526870fcff106c7ff7ac19c55a/521ae416df386b804a8c63fde7d0580ea83db7526870fcff106c7ff7ac19c55a-json.log",
"Name": "/old-k8s-version-331000",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro",
"old-k8s-version-331000:/var"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "old-k8s-version-331000",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 2306867200,
"NanoCpus": 2000000000,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 2306867200,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/07526d52e2e1daf22fd7ffad3c7b505b9063cf2864df54dcbbafbdfe1d92fc29-init/diff:/var/lib/docker/overlay2/9fbf3b44ff597f1722ec4c360ffdfcae800e09c24268f5f936302ffdfee42d32/diff:/var/lib/docker/overlay2/2fa58e45c276e0be77bbac970263b15f9ad2001b415cd256f2ba0fa98573e87f/diff:/var/lib/docker/overlay2/13592448013aa7997afb90ea8f709060aa47b73f59c20b387deeca946f5e5266/diff:/var/lib/docker/overlay2/9c9eab98a913029a905ee9043d4ea3ee591c2a3aeae75270696cf2f4b02139ff/diff:/var/lib/docker/overlay2/8369b3f99543b36671d6b1cad98dc7941e80cfbb758b02cee4ada4b05bcb4f0b/diff:/var/lib/docker/overlay2/c51ae82d21d70c3b9ce7715bc6a6a55190189c8010c12659f5e6b4fc071fcf51/diff:/var/lib/docker/overlay2/1cfa501ade9e79b858d1178588e001d5cdbedf2827ba12c92bc46c08ea1d1930/diff:/var/lib/docker/overlay2/73958304c7245dc492f266b09a7ed783314d048ca147fa168947fb98ba88124e/diff:/var/lib/docker/overlay2/a86028ac7e2c414af738e790ed658ffe4d751344c6dc27966e786ec7cf7703c1/diff:/var/lib/docker/overlay2/8421b4
006e78f7e0ae1d1064a5f50418040f650718ed63b5049cac7acda8429b/diff:/var/lib/docker/overlay2/f1aa735ff48a6070c6673d0397f3d7e592ed5d0dd6b2bf65bb9f45567d8e76b1/diff:/var/lib/docker/overlay2/7f60399d9dd327abdaefda924e875ea54223eb844b6c5c6ace4792e3fc0498e1/diff:/var/lib/docker/overlay2/e569b0d3e8b9b5e71f31ee227f7b8d83da14355d5b880c0f7e477ecf18e360dd/diff:/var/lib/docker/overlay2/7a0fdb748888accee02d68f26ba7c8f11ed7e0b1727beb633a8a02d41c762b9a/diff:/var/lib/docker/overlay2/90bb359231d0667c6a684304863352970e3854a24adc01da7d96250078f50593/diff:/var/lib/docker/overlay2/14807dd345c571446606fc12542343343734f09216ac0f72ea971e44c51e73bc/diff:/var/lib/docker/overlay2/47675c305fd3157c7d1f94df59badbf171128c371f27df3e460e7616fdebd1a3/diff:/var/lib/docker/overlay2/62eb07949377c89c1ed888624c8d9a730d01e2eb1dd06ff23fc64e28e0cbbc8c/diff:/var/lib/docker/overlay2/a4cd0c6fe63f05907e27cba37dcb62ac7283f4f6f3b3fbfda934216663875735/diff:/var/lib/docker/overlay2/a087ac93bbe33796bd250ecb00de5612af3696f1968d30c9b2390178f17f203b/diff:/var/lib/d
ocker/overlay2/56c471e8e71e11a4ea8a6a241a2406101bce3170edbec4f49d26e9d56894affa/diff:/var/lib/docker/overlay2/7bd516e66d75f51ad75387f0e86ddb6cd9e975fd71c57c5b2dcddf88ad78acd7/diff:/var/lib/docker/overlay2/20fca7abbce0276665d7149fd104aaff2fbce77f78ab0b4ced10bca46257d41c/diff:/var/lib/docker/overlay2/5e10b3d425507f2bf83efc712253364a52c3efa222e08dc12d7ed04e6390fbf8/diff:/var/lib/docker/overlay2/e8efccb1871a01c5184fe7925e831ce2fd83d4f03065027be2901b92ab90a258/diff:/var/lib/docker/overlay2/97780e9d41ac53335964c277f20f6ec8a960ee705fbeb461d6c712bde94de436/diff:/var/lib/docker/overlay2/f72e5826af1361739ac8503fbb88bf39f8efc9ffbb41af7012e3aa815236ed48/diff:/var/lib/docker/overlay2/a40d3ff72764915cfa40dfd926b8f13684764d6c58dd0a3ad4301584bb0998f3/diff:/var/lib/docker/overlay2/5730e50e96fd55a5c5fa6a2d4ebd253a91e71d32989764a774722d2703760ef0/diff:/var/lib/docker/overlay2/b9c7c3ad8aee072ae5cb7e36227bb14403f096a48ad1140bd0c499590859c6cc/diff:/var/lib/docker/overlay2/e66abe381c2911f74ec47698d1bdecfd39268170629b92fd84b37eb2ce7
a5ba6/diff:/var/lib/docker/overlay2/808d85518d0d508262fe102c8c8ada39430d040fe587d8f352d242a7cb41c3d6/diff:/var/lib/docker/overlay2/415facd46666ec88fe94221167e53a0e81eb9d72fef6326abf1a5a62bd28f6a0/diff:/var/lib/docker/overlay2/4d191b6478677781e2f1e74ac65d7957f5b20b908733d882a049a62c2d8f6728/diff:/var/lib/docker/overlay2/76d1a08b1a9c336558d2ef846ee06a10bc3513ce602ecf525e72588e05cee440/diff:/var/lib/docker/overlay2/917a1b8e5732828fa00857e6bc4b53263c418e42154740f628e7ba65ee6296a3/diff:/var/lib/docker/overlay2/80c17b561fc7c2c0512663a0616a853c94e8ca142c247f82e6b22e957d2853ba/diff:/var/lib/docker/overlay2/9b2baf8bcd946ef6986df8016336a5290e46a343ba3a203b45b5eaccf2d19bdf/diff:/var/lib/docker/overlay2/32b8331b04aa0b9282abe09a864255a764f85400b86fb4dbe23f9494e0c3a93c/diff:/var/lib/docker/overlay2/18634eb2fa4552ab5e8a5683b2d1d519cf3ca8fb3d99d345219c89f4d93494b3/diff:/var/lib/docker/overlay2/5bbe864c28792b9309fdb4840452e1a20c13632cf0560091a4204b8ad0c246d3/diff:/var/lib/docker/overlay2/c5589da650ee6241c9902fbf401eaf68da7b60
ab69831f2e9c321d744113f417/diff",
"MergedDir": "/var/lib/docker/overlay2/07526d52e2e1daf22fd7ffad3c7b505b9063cf2864df54dcbbafbdfe1d92fc29/merged",
"UpperDir": "/var/lib/docker/overlay2/07526d52e2e1daf22fd7ffad3c7b505b9063cf2864df54dcbbafbdfe1d92fc29/diff",
"WorkDir": "/var/lib/docker/overlay2/07526d52e2e1daf22fd7ffad3c7b505b9063cf2864df54dcbbafbdfe1d92fc29/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "volume",
"Name": "old-k8s-version-331000",
"Source": "/var/lib/docker/volumes/old-k8s-version-331000/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
},
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "old-k8s-version-331000",
"Domainname": "",
"User": "root",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"22/tcp": {},
"2376/tcp": {},
"32443/tcp": {},
"5000/tcp": {},
"8443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase-builds:v0.0.37-1679075007-16079@sha256:9d2bf610e530f47c7e43df9719717075e4c7ec510aea6047b29208bffe7f1978",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "old-k8s-version-331000",
"name.minikube.sigs.k8s.io": "old-k8s-version-331000",
"org.opencontainers.image.ref.name": "ubuntu",
"org.opencontainers.image.version": "20.04",
"role.minikube.sigs.k8s.io": ""
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "1c3400d6614f1fcfc40438330413bf45914c50ed0695474399ec3c8f922241a0",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "58897"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "58898"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "58899"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "58900"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "58901"
}
]
},
"SandboxKey": "/var/run/docker/netns/1c3400d6614f",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"old-k8s-version-331000": {
"IPAMConfig": {
"IPv4Address": "192.168.76.2"
},
"Links": null,
"Aliases": [
"521ae416df38",
"old-k8s-version-331000"
],
"NetworkID": "6fa3318f5503a289e050622f1a9f4b0bea1af814ad18d1b409b3b8c51f5769ef",
"EndpointID": "aece95ad704ac274d6413327b529a3749e8ce01de608325aac7061bad1ca54c9",
"Gateway": "192.168.76.1",
"IPAddress": "192.168.76.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:c0:a8:4c:02",
"DriverOpts": null
}
}
}
}
]
-- /stdout --
helpers_test.go:239: (dbg) Run: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-331000 -n old-k8s-version-331000
helpers_test.go:239: (dbg) Non-zero exit: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-331000 -n old-k8s-version-331000: exit status 6 (400.889411ms)
-- stdout --
Running
WARNING: Your kubectl is pointing to stale minikube-vm.
To fix the kubectl context, run `minikube update-context`
-- /stdout --
** stderr **
E0330 09:33:29.678407 43874 status.go:415] kubeconfig endpoint: extract IP: "old-k8s-version-331000" does not appear in /Users/jenkins/minikube-integration/16199-24978/kubeconfig
** /stderr **
helpers_test.go:239: status error: exit status 6 (may be ok)
helpers_test.go:241: "old-k8s-version-331000" host is not running, skipping log retrieval (state="Running\nWARNING: Your kubectl is pointing to stale minikube-vm.\nTo fix the kubectl context, run `minikube update-context`")
--- FAIL: TestStartStop/group/old-k8s-version/serial/FirstStart (252.50s)