=== RUN TestStartStop/group/old-k8s-version/serial/FirstStart
start_stop_delete_test.go:186: (dbg) Run: out/minikube-darwin-amd64 start -p old-k8s-version-770000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0
start_stop_delete_test.go:186: (dbg) Non-zero exit: out/minikube-darwin-amd64 start -p old-k8s-version-770000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0: exit status 109 (4m17.679612309s)
-- stdout --
* [old-k8s-version-770000] minikube v1.31.0 on Darwin 13.4.1
- MINIKUBE_LOCATION=16899
- KUBECONFIG=/Users/jenkins/minikube-integration/16899-76867/kubeconfig
- MINIKUBE_BIN=out/minikube-darwin-amd64
- MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
- MINIKUBE_HOME=/Users/jenkins/minikube-integration/16899-76867/.minikube
- MINIKUBE_FORCE_SYSTEMD=
* Using the docker driver based on user configuration
* Using Docker Desktop driver with root privileges
* Starting control plane node old-k8s-version-770000 in cluster old-k8s-version-770000
* Pulling base image ...
* Creating docker container (CPUs=2, Memory=2200MB) ...
* Preparing Kubernetes v1.16.0 on Docker 24.0.4 ...
- Generating certificates and keys ...
- Booting up control plane ...
- Generating certificates and keys ...
- Booting up control plane ...
-- /stdout --
** stderr **
I0717 16:01:24.542673 92606 out.go:296] Setting OutFile to fd 1 ...
I0717 16:01:24.542862 92606 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0717 16:01:24.542868 92606 out.go:309] Setting ErrFile to fd 2...
I0717 16:01:24.542873 92606 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0717 16:01:24.543050 92606 root.go:338] Updating PATH: /Users/jenkins/minikube-integration/16899-76867/.minikube/bin
I0717 16:01:24.544603 92606 out.go:303] Setting JSON to false
I0717 16:01:24.564344 92606 start.go:128] hostinfo: {"hostname":"MacOS-Agent-1.local","uptime":25252,"bootTime":1689609632,"procs":439,"os":"darwin","platform":"darwin","platformFamily":"Standalone Workstation","platformVersion":"13.4.1","kernelVersion":"22.5.0","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"b7610dcb-1435-5842-8d5a-b2388403fea3"}
W0717 16:01:24.564429 92606 start.go:136] gopshost.Virtualization returned error: not implemented yet
I0717 16:01:24.586979 92606 out.go:177] * [old-k8s-version-770000] minikube v1.31.0 on Darwin 13.4.1
I0717 16:01:24.628876 92606 notify.go:220] Checking for updates...
I0717 16:01:24.628900 92606 out.go:177] - MINIKUBE_LOCATION=16899
I0717 16:01:24.672058 92606 out.go:177] - KUBECONFIG=/Users/jenkins/minikube-integration/16899-76867/kubeconfig
I0717 16:01:24.692709 92606 out.go:177] - MINIKUBE_BIN=out/minikube-darwin-amd64
I0717 16:01:24.713942 92606 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0717 16:01:24.735096 92606 out.go:177] - MINIKUBE_HOME=/Users/jenkins/minikube-integration/16899-76867/.minikube
I0717 16:01:24.755674 92606 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0717 16:01:24.777629 92606 config.go:182] Loaded profile config "calico-679000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.27.3
I0717 16:01:24.777802 92606 driver.go:373] Setting default libvirt URI to qemu:///system
I0717 16:01:24.834581 92606 docker.go:121] docker version: linux-24.0.2:Docker Desktop 4.21.1 (114176)
I0717 16:01:24.834711 92606 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0717 16:01:24.937600 92606 info.go:266] docker info: {ID:37b3081a-8cd6-457e-b2a4-79dc82345b06 Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:55 OomKillDisable:false NGoroutines:68 SystemTime:2023-07-17 23:01:24.924955543 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:8 KernelVersion:5.15.49-linuxkit-pr OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexS
erverAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:6 MemTotal:6231715840 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:24.0.2 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:3dce8eb055cbb6872793272b4f20ed16117344f8 Expected:3dce8eb055cbb6872793272b4f20ed16117344f8} RuncCommit:{ID:v1.1.7-0-g860f061 Expected:v1.1.7-0-g860f061} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=built
in name=cgroupns] ProductLicense: Warnings:<nil> ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.11.0] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.19.1] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker Dev Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Manages Docker extensions Vendor:Docker Inc. Version:v0.2.20] map[Name:init Pat
h:/Users/jenkins/.docker/cli-plugins/docker-init SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-init] ShortDescription:Creates Docker-related starter files for your project Vendor:Docker Inc. Version:v0.1.0-beta.6] map[Name:sbom Path:/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM) for an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scan] ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.26.0] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Command line tool for Docker Scout Vendor:Docker Inc. Version:0.16.1]] Warnings:<n
il>}}
I0717 16:01:24.979685 92606 out.go:177] * Using the docker driver based on user configuration
I0717 16:01:25.000705 92606 start.go:298] selected driver: docker
I0717 16:01:25.000731 92606 start.go:880] validating driver "docker" against <nil>
I0717 16:01:25.000747 92606 start.go:891] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0717 16:01:25.004481 92606 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0717 16:01:25.107864 92606 info.go:266] docker info: {ID:37b3081a-8cd6-457e-b2a4-79dc82345b06 Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:55 OomKillDisable:false NGoroutines:68 SystemTime:2023-07-17 23:01:25.095953316 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:8 KernelVersion:5.15.49-linuxkit-pr OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexS
erverAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:6 MemTotal:6231715840 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:24.0.2 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:3dce8eb055cbb6872793272b4f20ed16117344f8 Expected:3dce8eb055cbb6872793272b4f20ed16117344f8} RuncCommit:{ID:v1.1.7-0-g860f061 Expected:v1.1.7-0-g860f061} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=built
in name=cgroupns] ProductLicense: Warnings:<nil> ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.11.0] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.19.1] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker Dev Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Manages Docker extensions Vendor:Docker Inc. Version:v0.2.20] map[Name:init Pat
h:/Users/jenkins/.docker/cli-plugins/docker-init SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-init] ShortDescription:Creates Docker-related starter files for your project Vendor:Docker Inc. Version:v0.1.0-beta.6] map[Name:sbom Path:/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM) for an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scan] ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.26.0] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Command line tool for Docker Scout Vendor:Docker Inc. Version:0.16.1]] Warnings:<n
il>}}
I0717 16:01:25.108043 92606 start_flags.go:305] no existing cluster config was found, will generate one from the flags
I0717 16:01:25.108256 92606 start_flags.go:919] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0717 16:01:25.129913 92606 out.go:177] * Using Docker Desktop driver with root privileges
I0717 16:01:25.150473 92606 cni.go:84] Creating CNI manager for ""
I0717 16:01:25.150511 92606 cni.go:168] CNI unnecessary in this configuration, recommending no CNI
I0717 16:01:25.150529 92606 start_flags.go:319] config:
{Name:old-k8s-version-770000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-770000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocke
t: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0}
I0717 16:01:25.193619 92606 out.go:177] * Starting control plane node old-k8s-version-770000 in cluster old-k8s-version-770000
I0717 16:01:25.214729 92606 cache.go:122] Beginning downloading kic base image for docker with docker
I0717 16:01:25.236690 92606 out.go:177] * Pulling base image ...
I0717 16:01:25.294716 92606 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0717 16:01:25.294776 92606 image.go:79] Checking for gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 in local docker daemon
I0717 16:01:25.294834 92606 preload.go:148] Found local preload: /Users/jenkins/minikube-integration/16899-76867/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4
I0717 16:01:25.294861 92606 cache.go:57] Caching tarball of preloaded images
I0717 16:01:25.295108 92606 preload.go:174] Found /Users/jenkins/minikube-integration/16899-76867/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0717 16:01:25.295138 92606 cache.go:60] Finished verifying existence of preloaded tar for v1.16.0 on docker
I0717 16:01:25.296233 92606 profile.go:148] Saving config to /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/config.json ...
I0717 16:01:25.296439 92606 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/config.json: {Name:mkff572bcce20dd159d8de0c14cbfe6bb39d473b Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0717 16:01:25.345673 92606 image.go:83] Found gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 in local docker daemon, skipping pull
I0717 16:01:25.345693 92606 cache.go:145] gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 exists in daemon, skipping load
I0717 16:01:25.345713 92606 cache.go:195] Successfully downloaded all kic artifacts
I0717 16:01:25.345752 92606 start.go:365] acquiring machines lock for old-k8s-version-770000: {Name:mk0f9163ab3562db295835b9e526369b56772523 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0717 16:01:25.345951 92606 start.go:369] acquired machines lock for "old-k8s-version-770000" in 186.726µs
I0717 16:01:25.345979 92606 start.go:93] Provisioning new machine with config: &{Name:old-k8s-version-770000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-770000 Namespace:default APIServerName:minikubeCA APIServe
rNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:fals
e CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0} &{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}
I0717 16:01:25.346074 92606 start.go:125] createHost starting for "" (driver="docker")
I0717 16:01:25.388813 92606 out.go:204] * Creating docker container (CPUs=2, Memory=2200MB) ...
I0717 16:01:25.389181 92606 start.go:159] libmachine.API.Create for "old-k8s-version-770000" (driver="docker")
I0717 16:01:25.389226 92606 client.go:168] LocalClient.Create starting
I0717 16:01:25.389424 92606 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/16899-76867/.minikube/certs/ca.pem
I0717 16:01:25.389497 92606 main.go:141] libmachine: Decoding PEM data...
I0717 16:01:25.389539 92606 main.go:141] libmachine: Parsing certificate...
I0717 16:01:25.389669 92606 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/16899-76867/.minikube/certs/cert.pem
I0717 16:01:25.389721 92606 main.go:141] libmachine: Decoding PEM data...
I0717 16:01:25.389738 92606 main.go:141] libmachine: Parsing certificate...
I0717 16:01:25.390653 92606 cli_runner.go:164] Run: docker network inspect old-k8s-version-770000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
W0717 16:01:25.439953 92606 cli_runner.go:211] docker network inspect old-k8s-version-770000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1
I0717 16:01:25.440057 92606 network_create.go:281] running [docker network inspect old-k8s-version-770000] to gather additional debugging logs...
I0717 16:01:25.440074 92606 cli_runner.go:164] Run: docker network inspect old-k8s-version-770000
W0717 16:01:25.491159 92606 cli_runner.go:211] docker network inspect old-k8s-version-770000 returned with exit code 1
I0717 16:01:25.491234 92606 network_create.go:284] error running [docker network inspect old-k8s-version-770000]: docker network inspect old-k8s-version-770000: exit status 1
stdout:
[]
stderr:
Error response from daemon: network old-k8s-version-770000 not found
I0717 16:01:25.491250 92606 network_create.go:286] output of [docker network inspect old-k8s-version-770000]: -- stdout --
[]
-- /stdout --
** stderr **
Error response from daemon: network old-k8s-version-770000 not found
** /stderr **
I0717 16:01:25.491370 92606 cli_runner.go:164] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0717 16:01:25.544461 92606 network.go:212] skipping subnet 192.168.49.0/24 that is reserved: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0717 16:01:25.544828 92606 network.go:209] using free private subnet 192.168.58.0/24: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc000dc07c0}
I0717 16:01:25.544849 92606 network_create.go:123] attempt to create docker network old-k8s-version-770000 192.168.58.0/24 with gateway 192.168.58.1 and MTU of 65535 ...
I0717 16:01:25.544928 92606 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.58.0/24 --gateway=192.168.58.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-770000 old-k8s-version-770000
W0717 16:01:25.594696 92606 cli_runner.go:211] docker network create --driver=bridge --subnet=192.168.58.0/24 --gateway=192.168.58.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-770000 old-k8s-version-770000 returned with exit code 1
W0717 16:01:25.594728 92606 network_create.go:148] failed to create docker network old-k8s-version-770000 192.168.58.0/24 with gateway 192.168.58.1 and mtu of 65535: docker network create --driver=bridge --subnet=192.168.58.0/24 --gateway=192.168.58.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-770000 old-k8s-version-770000: exit status 1
stdout:
stderr:
Error response from daemon: Pool overlaps with other one on this address space
W0717 16:01:25.594747 92606 network_create.go:115] failed to create docker network old-k8s-version-770000 192.168.58.0/24, will retry: subnet is taken
I0717 16:01:25.596081 92606 network.go:212] skipping subnet 192.168.58.0/24 that is reserved: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0717 16:01:25.596399 92606 network.go:209] using free private subnet 192.168.67.0/24: &{IP:192.168.67.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.67.0/24 Gateway:192.168.67.1 ClientMin:192.168.67.2 ClientMax:192.168.67.254 Broadcast:192.168.67.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc00070f800}
I0717 16:01:25.596412 92606 network_create.go:123] attempt to create docker network old-k8s-version-770000 192.168.67.0/24 with gateway 192.168.67.1 and MTU of 65535 ...
I0717 16:01:25.596477 92606 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.67.0/24 --gateway=192.168.67.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-770000 old-k8s-version-770000
W0717 16:01:25.646102 92606 cli_runner.go:211] docker network create --driver=bridge --subnet=192.168.67.0/24 --gateway=192.168.67.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-770000 old-k8s-version-770000 returned with exit code 1
W0717 16:01:25.646141 92606 network_create.go:148] failed to create docker network old-k8s-version-770000 192.168.67.0/24 with gateway 192.168.67.1 and mtu of 65535: docker network create --driver=bridge --subnet=192.168.67.0/24 --gateway=192.168.67.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-770000 old-k8s-version-770000: exit status 1
stdout:
stderr:
Error response from daemon: Pool overlaps with other one on this address space
W0717 16:01:25.646157 92606 network_create.go:115] failed to create docker network old-k8s-version-770000 192.168.67.0/24, will retry: subnet is taken
I0717 16:01:25.647727 92606 network.go:212] skipping subnet 192.168.67.0/24 that is reserved: &{IP:192.168.67.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.67.0/24 Gateway:192.168.67.1 ClientMin:192.168.67.2 ClientMax:192.168.67.254 Broadcast:192.168.67.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0717 16:01:25.648052 92606 network.go:209] using free private subnet 192.168.76.0/24: &{IP:192.168.76.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.76.0/24 Gateway:192.168.76.1 ClientMin:192.168.76.2 ClientMax:192.168.76.254 Broadcast:192.168.76.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc000e8df40}
I0717 16:01:25.648069 92606 network_create.go:123] attempt to create docker network old-k8s-version-770000 192.168.76.0/24 with gateway 192.168.76.1 and MTU of 65535 ...
I0717 16:01:25.648134 92606 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.76.0/24 --gateway=192.168.76.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-770000 old-k8s-version-770000
I0717 16:01:25.730400 92606 network_create.go:107] docker network old-k8s-version-770000 192.168.76.0/24 created
I0717 16:01:25.730447 92606 kic.go:117] calculated static IP "192.168.76.2" for the "old-k8s-version-770000" container
I0717 16:01:25.730574 92606 cli_runner.go:164] Run: docker ps -a --format {{.Names}}
I0717 16:01:25.780443 92606 cli_runner.go:164] Run: docker volume create old-k8s-version-770000 --label name.minikube.sigs.k8s.io=old-k8s-version-770000 --label created_by.minikube.sigs.k8s.io=true
I0717 16:01:25.830567 92606 oci.go:103] Successfully created a docker volume old-k8s-version-770000
I0717 16:01:25.830689 92606 cli_runner.go:164] Run: docker run --rm --name old-k8s-version-770000-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-770000 --entrypoint /usr/bin/test -v old-k8s-version-770000:/var gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 -d /var/lib
I0717 16:01:26.326123 92606 oci.go:107] Successfully prepared a docker volume old-k8s-version-770000
I0717 16:01:26.326162 92606 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0717 16:01:26.326178 92606 kic.go:190] Starting extracting preloaded images to volume ...
I0717 16:01:26.326296 92606 cli_runner.go:164] Run: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/16899-76867/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-770000:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 -I lz4 -xf /preloaded.tar -C /extractDir
I0717 16:01:29.146536 92606 cli_runner.go:217] Completed: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/16899-76867/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-770000:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 -I lz4 -xf /preloaded.tar -C /extractDir: (2.820027546s)
I0717 16:01:29.146566 92606 kic.go:199] duration metric: took 2.820233 seconds to extract preloaded images to volume
I0717 16:01:29.146679 92606 cli_runner.go:164] Run: docker info --format "'{{json .SecurityOptions}}'"
I0717 16:01:29.248415 92606 cli_runner.go:164] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname old-k8s-version-770000 --name old-k8s-version-770000 --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-770000 --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=old-k8s-version-770000 --network old-k8s-version-770000 --ip 192.168.76.2 --volume old-k8s-version-770000:/var --security-opt apparmor=unconfined --memory=2200mb --memory-swap=2200mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631
I0717 16:01:29.526563 92606 cli_runner.go:164] Run: docker container inspect old-k8s-version-770000 --format={{.State.Running}}
I0717 16:01:29.585979 92606 cli_runner.go:164] Run: docker container inspect old-k8s-version-770000 --format={{.State.Status}}
I0717 16:01:29.641218 92606 cli_runner.go:164] Run: docker exec old-k8s-version-770000 stat /var/lib/dpkg/alternatives/iptables
I0717 16:01:29.741999 92606 oci.go:144] the created container "old-k8s-version-770000" has a running status.
I0717 16:01:29.742048 92606 kic.go:221] Creating ssh key for kic: /Users/jenkins/minikube-integration/16899-76867/.minikube/machines/old-k8s-version-770000/id_rsa...
I0717 16:01:29.863646 92606 kic_runner.go:191] docker (temp): /Users/jenkins/minikube-integration/16899-76867/.minikube/machines/old-k8s-version-770000/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0717 16:01:29.931585 92606 cli_runner.go:164] Run: docker container inspect old-k8s-version-770000 --format={{.State.Status}}
I0717 16:01:29.984971 92606 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0717 16:01:29.984993 92606 kic_runner.go:114] Args: [docker exec --privileged old-k8s-version-770000 chown docker:docker /home/docker/.ssh/authorized_keys]
I0717 16:01:30.083316 92606 cli_runner.go:164] Run: docker container inspect old-k8s-version-770000 --format={{.State.Status}}
I0717 16:01:30.137601 92606 machine.go:88] provisioning docker machine ...
I0717 16:01:30.137647 92606 ubuntu.go:169] provisioning hostname "old-k8s-version-770000"
I0717 16:01:30.137756 92606 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-770000
I0717 16:01:30.191940 92606 main.go:141] libmachine: Using SSH client type: native
I0717 16:01:30.192336 92606 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x140d2c0] 0x1410360 <nil> [] 0s} 127.0.0.1 57123 <nil> <nil>}
I0717 16:01:30.192347 92606 main.go:141] libmachine: About to run SSH command:
sudo hostname old-k8s-version-770000 && echo "old-k8s-version-770000" | sudo tee /etc/hostname
I0717 16:01:30.333426 92606 main.go:141] libmachine: SSH cmd err, output: <nil>: old-k8s-version-770000
I0717 16:01:30.333514 92606 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-770000
I0717 16:01:30.385244 92606 main.go:141] libmachine: Using SSH client type: native
I0717 16:01:30.385586 92606 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x140d2c0] 0x1410360 <nil> [] 0s} 127.0.0.1 57123 <nil> <nil>}
I0717 16:01:30.385601 92606 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sold-k8s-version-770000' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 old-k8s-version-770000/g' /etc/hosts;
else
echo '127.0.1.1 old-k8s-version-770000' | sudo tee -a /etc/hosts;
fi
fi
I0717 16:01:30.513862 92606 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0717 16:01:30.513885 92606 ubuntu.go:175] set auth options {CertDir:/Users/jenkins/minikube-integration/16899-76867/.minikube CaCertPath:/Users/jenkins/minikube-integration/16899-76867/.minikube/certs/ca.pem CaPrivateKeyPath:/Users/jenkins/minikube-integration/16899-76867/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/Users/jenkins/minikube-integration/16899-76867/.minikube/machines/server.pem ServerKeyPath:/Users/jenkins/minikube-integration/16899-76867/.minikube/machines/server-key.pem ClientKeyPath:/Users/jenkins/minikube-integration/16899-76867/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/Users/jenkins/minikube-integration/16899-76867/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/Users/jenkins/minikube-integration/16899-76867/.minikube}
I0717 16:01:30.513914 92606 ubuntu.go:177] setting up certificates
I0717 16:01:30.513924 92606 provision.go:83] configureAuth start
I0717 16:01:30.514006 92606 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-770000
I0717 16:01:30.564855 92606 provision.go:138] copyHostCerts
I0717 16:01:30.564967 92606 exec_runner.go:144] found /Users/jenkins/minikube-integration/16899-76867/.minikube/key.pem, removing ...
I0717 16:01:30.564976 92606 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/16899-76867/.minikube/key.pem
I0717 16:01:30.565103 92606 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/16899-76867/.minikube/certs/key.pem --> /Users/jenkins/minikube-integration/16899-76867/.minikube/key.pem (1675 bytes)
I0717 16:01:30.565312 92606 exec_runner.go:144] found /Users/jenkins/minikube-integration/16899-76867/.minikube/ca.pem, removing ...
I0717 16:01:30.565318 92606 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/16899-76867/.minikube/ca.pem
I0717 16:01:30.565387 92606 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/16899-76867/.minikube/certs/ca.pem --> /Users/jenkins/minikube-integration/16899-76867/.minikube/ca.pem (1078 bytes)
I0717 16:01:30.565535 92606 exec_runner.go:144] found /Users/jenkins/minikube-integration/16899-76867/.minikube/cert.pem, removing ...
I0717 16:01:30.565541 92606 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/16899-76867/.minikube/cert.pem
I0717 16:01:30.565661 92606 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/16899-76867/.minikube/certs/cert.pem --> /Users/jenkins/minikube-integration/16899-76867/.minikube/cert.pem (1123 bytes)
I0717 16:01:30.565791 92606 provision.go:112] generating server cert: /Users/jenkins/minikube-integration/16899-76867/.minikube/machines/server.pem ca-key=/Users/jenkins/minikube-integration/16899-76867/.minikube/certs/ca.pem private-key=/Users/jenkins/minikube-integration/16899-76867/.minikube/certs/ca-key.pem org=jenkins.old-k8s-version-770000 san=[192.168.76.2 127.0.0.1 localhost 127.0.0.1 minikube old-k8s-version-770000]
I0717 16:01:30.689553 92606 provision.go:172] copyRemoteCerts
I0717 16:01:30.689664 92606 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0717 16:01:30.689714 92606 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-770000
I0717 16:01:30.743149 92606 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:57123 SSHKeyPath:/Users/jenkins/minikube-integration/16899-76867/.minikube/machines/old-k8s-version-770000/id_rsa Username:docker}
I0717 16:01:30.837825 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1078 bytes)
I0717 16:01:30.860510 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/machines/server.pem --> /etc/docker/server.pem (1241 bytes)
I0717 16:01:30.882397 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1679 bytes)
I0717 16:01:30.904603 92606 provision.go:86] duration metric: configureAuth took 390.645426ms
I0717 16:01:30.904618 92606 ubuntu.go:193] setting minikube options for container-runtime
I0717 16:01:30.904766 92606 config.go:182] Loaded profile config "old-k8s-version-770000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.16.0
I0717 16:01:30.904874 92606 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-770000
I0717 16:01:30.957345 92606 main.go:141] libmachine: Using SSH client type: native
I0717 16:01:30.957706 92606 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x140d2c0] 0x1410360 <nil> [] 0s} 127.0.0.1 57123 <nil> <nil>}
I0717 16:01:30.957723 92606 main.go:141] libmachine: About to run SSH command:
df --output=fstype / | tail -n 1
I0717 16:01:31.086411 92606 main.go:141] libmachine: SSH cmd err, output: <nil>: overlay
I0717 16:01:31.086436 92606 ubuntu.go:71] root file system type: overlay
I0717 16:01:31.086527 92606 provision.go:309] Updating docker unit: /lib/systemd/system/docker.service ...
I0717 16:01:31.086621 92606 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-770000
I0717 16:01:31.137493 92606 main.go:141] libmachine: Using SSH client type: native
I0717 16:01:31.137846 92606 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x140d2c0] 0x1410360 <nil> [] 0s} 127.0.0.1 57123 <nil> <nil>}
I0717 16:01:31.137898 92606 main.go:141] libmachine: About to run SSH command:
sudo mkdir -p /lib/systemd/system && printf %s "[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP \$MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service.new
I0717 16:01:31.276827 92606 main.go:141] libmachine: SSH cmd err, output: <nil>: [Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
I0717 16:01:31.276933 92606 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-770000
I0717 16:01:31.328350 92606 main.go:141] libmachine: Using SSH client type: native
I0717 16:01:31.328708 92606 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x140d2c0] 0x1410360 <nil> [] 0s} 127.0.0.1 57123 <nil> <nil>}
I0717 16:01:31.328722 92606 main.go:141] libmachine: About to run SSH command:
sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
I0717 16:01:31.993924 92606 main.go:141] libmachine: SSH cmd err, output: <nil>: --- /lib/systemd/system/docker.service 2023-07-07 14:50:55.000000000 +0000
+++ /lib/systemd/system/docker.service.new 2023-07-17 23:01:31.273481749 +0000
@@ -1,30 +1,32 @@
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
-After=network-online.target docker.socket firewalld.service containerd.service time-set.target
-Wants=network-online.target containerd.service
+BindsTo=containerd.service
+After=network-online.target firewalld.service containerd.service
+Wants=network-online.target
Requires=docker.socket
+StartLimitBurst=3
+StartLimitIntervalSec=60
[Service]
Type=notify
-# the default is not to use systemd for cgroups because the delegate issues still
-# exists and systemd currently does not support the cgroup feature set required
-# for containers run by docker
-ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
-ExecReload=/bin/kill -s HUP $MAINPID
-TimeoutStartSec=0
-RestartSec=2
-Restart=always
+Restart=on-failure
-# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
-# Both the old, and new location are accepted by systemd 229 and up, so using the old location
-# to make them work for either version of systemd.
-StartLimitBurst=3
-# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
-# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
-# this option work for either version of systemd.
-StartLimitInterval=60s
+
+# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
+# The base configuration already specifies an 'ExecStart=...' command. The first directive
+# here is to clear out that command inherited from the base configuration. Without this,
+# the command from the base configuration and the command specified here are treated as
+# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
+# will catch this invalid input and refuse to start the service with an error like:
+# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
+
+# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
+# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
+ExecStart=
+ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
+ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
@@ -32,16 +34,16 @@
LimitNPROC=infinity
LimitCORE=infinity
-# Comment TasksMax if your systemd version does not support it.
-# Only systemd 226 and above support this option.
+# Uncomment TasksMax if your systemd version supports it.
+# Only systemd 226 and above support this version.
TasksMax=infinity
+TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
-OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
I0717 16:01:31.993948 92606 machine.go:91] provisioned docker machine in 1.85624136s
I0717 16:01:31.993962 92606 client.go:171] LocalClient.Create took 6.604384544s
I0717 16:01:31.993987 92606 start.go:167] duration metric: libmachine.API.Create for "old-k8s-version-770000" took 6.604464031s
I0717 16:01:31.994014 92606 start.go:300] post-start starting for "old-k8s-version-770000" (driver="docker")
I0717 16:01:31.994027 92606 start.go:329] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0717 16:01:31.994097 92606 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0717 16:01:31.994160 92606 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-770000
I0717 16:01:32.045886 92606 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:57123 SSHKeyPath:/Users/jenkins/minikube-integration/16899-76867/.minikube/machines/old-k8s-version-770000/id_rsa Username:docker}
I0717 16:01:32.138516 92606 ssh_runner.go:195] Run: cat /etc/os-release
I0717 16:01:32.142692 92606 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0717 16:01:32.142714 92606 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0717 16:01:32.142722 92606 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0717 16:01:32.142727 92606 info.go:137] Remote host: Ubuntu 22.04.2 LTS
I0717 16:01:32.142736 92606 filesync.go:126] Scanning /Users/jenkins/minikube-integration/16899-76867/.minikube/addons for local assets ...
I0717 16:01:32.142821 92606 filesync.go:126] Scanning /Users/jenkins/minikube-integration/16899-76867/.minikube/files for local assets ...
I0717 16:01:32.143002 92606 filesync.go:149] local asset: /Users/jenkins/minikube-integration/16899-76867/.minikube/files/etc/ssl/certs/773242.pem -> 773242.pem in /etc/ssl/certs
I0717 16:01:32.143182 92606 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0717 16:01:32.152208 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/files/etc/ssl/certs/773242.pem --> /etc/ssl/certs/773242.pem (1708 bytes)
I0717 16:01:32.174511 92606 start.go:303] post-start completed in 180.476259ms
I0717 16:01:32.175045 92606 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-770000
I0717 16:01:32.227654 92606 profile.go:148] Saving config to /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/config.json ...
I0717 16:01:32.228094 92606 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0717 16:01:32.228156 92606 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-770000
I0717 16:01:32.278725 92606 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:57123 SSHKeyPath:/Users/jenkins/minikube-integration/16899-76867/.minikube/machines/old-k8s-version-770000/id_rsa Username:docker}
I0717 16:01:32.368244 92606 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0717 16:01:32.373504 92606 start.go:128] duration metric: createHost completed in 7.027054995s
I0717 16:01:32.373521 92606 start.go:83] releasing machines lock for "old-k8s-version-770000", held for 7.027198537s
I0717 16:01:32.373626 92606 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-770000
I0717 16:01:32.424854 92606 ssh_runner.go:195] Run: cat /version.json
I0717 16:01:32.424873 92606 ssh_runner.go:195] Run: curl -sS -m 2 https://registry.k8s.io/
I0717 16:01:32.424936 92606 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-770000
I0717 16:01:32.424953 92606 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-770000
I0717 16:01:32.481001 92606 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:57123 SSHKeyPath:/Users/jenkins/minikube-integration/16899-76867/.minikube/machines/old-k8s-version-770000/id_rsa Username:docker}
I0717 16:01:32.484890 92606 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:57123 SSHKeyPath:/Users/jenkins/minikube-integration/16899-76867/.minikube/machines/old-k8s-version-770000/id_rsa Username:docker}
I0717 16:01:32.677685 92606 ssh_runner.go:195] Run: systemctl --version
I0717 16:01:32.683121 92606 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
I0717 16:01:32.688598 92606 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *loopback.conf* -not -name *.mk_disabled -exec sh -c "grep -q loopback {} && ( grep -q name {} || sudo sed -i '/"type": "loopback"/i \ \ \ \ "name": "loopback",' {} ) && sudo sed -i 's|"cniVersion": ".*"|"cniVersion": "1.0.0"|g' {}" ;
I0717 16:01:32.713113 92606 cni.go:236] loopback cni configuration patched: "/etc/cni/net.d/*loopback.conf*" found
I0717 16:01:32.713177 92606 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *bridge* -not -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e '/"dst": ".*:.*"/d' -e 's|^(.*)"dst": (.*)[,*]$|\1"dst": \2|g' -e '/"subnet": ".*:.*"/d' -e 's|^(.*)"subnet": ".*"(.*)[,*]$|\1"subnet": "10.244.0.0/16"\2|g' {}" ;
I0717 16:01:32.729607 92606 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e 's|^(.*)"subnet": ".*"(.*)$|\1"subnet": "10.244.0.0/16"\2|g' -e 's|^(.*)"gateway": ".*"(.*)$|\1"gateway": "10.244.0.1"\2|g' {}" ;
I0717 16:01:32.745444 92606 cni.go:314] configured [/etc/cni/net.d/100-crio-bridge.conf, /etc/cni/net.d/87-podman-bridge.conflist] bridge cni config(s)
I0717 16:01:32.745462 92606 start.go:466] detecting cgroup driver to use...
I0717 16:01:32.745477 92606 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0717 16:01:32.745593 92606 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0717 16:01:32.761564 92606 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.1"|' /etc/containerd/config.toml"
I0717 16:01:32.771578 92606 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I0717 16:01:32.781421 92606 containerd.go:145] configuring containerd to use "cgroupfs" as cgroup driver...
I0717 16:01:32.781481 92606 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = false|g' /etc/containerd/config.toml"
I0717 16:01:32.791352 92606 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0717 16:01:32.801376 92606 ssh_runner.go:195] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I0717 16:01:32.811460 92606 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0717 16:01:32.821149 92606 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0717 16:01:32.830538 92606 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I0717 16:01:32.840533 92606 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0717 16:01:32.849588 92606 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0717 16:01:32.857874 92606 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0717 16:01:32.932274 92606 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0717 16:01:33.002780 92606 start.go:466] detecting cgroup driver to use...
I0717 16:01:33.002801 92606 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0717 16:01:33.002872 92606 ssh_runner.go:195] Run: sudo systemctl cat docker.service
I0717 16:01:33.016008 92606 cruntime.go:276] skipping containerd shutdown because we are bound to it
I0717 16:01:33.016088 92606 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0717 16:01:33.028550 92606 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///var/run/dockershim.sock
" | sudo tee /etc/crictl.yaml"
I0717 16:01:33.046695 92606 ssh_runner.go:195] Run: which cri-dockerd
I0717 16:01:33.051698 92606 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/cri-docker.service.d
I0717 16:01:33.085814 92606 ssh_runner.go:362] scp memory --> /etc/systemd/system/cri-docker.service.d/10-cni.conf (189 bytes)
I0717 16:01:33.104660 92606 ssh_runner.go:195] Run: sudo systemctl unmask docker.service
I0717 16:01:33.205547 92606 ssh_runner.go:195] Run: sudo systemctl enable docker.socket
I0717 16:01:33.295720 92606 docker.go:535] configuring docker to use "cgroupfs" as cgroup driver...
I0717 16:01:33.295737 92606 ssh_runner.go:362] scp memory --> /etc/docker/daemon.json (144 bytes)
I0717 16:01:33.313589 92606 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0717 16:01:33.384030 92606 ssh_runner.go:195] Run: sudo systemctl restart docker
I0717 16:01:33.657882 92606 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0717 16:01:33.683993 92606 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0717 16:01:33.765797 92606 out.go:204] * Preparing Kubernetes v1.16.0 on Docker 24.0.4 ...
I0717 16:01:33.765885 92606 cli_runner.go:164] Run: docker exec -t old-k8s-version-770000 dig +short host.docker.internal
I0717 16:01:33.896251 92606 network.go:96] got host ip for mount in container by digging dns: 192.168.65.254
I0717 16:01:33.896391 92606 ssh_runner.go:195] Run: grep 192.168.65.254 host.minikube.internal$ /etc/hosts
I0717 16:01:33.902404 92606 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.65.254 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0717 16:01:33.917594 92606 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "8443/tcp") 0).HostPort}}'" old-k8s-version-770000
I0717 16:01:33.972987 92606 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0717 16:01:33.973064 92606 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0717 16:01:33.995367 92606 docker.go:636] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0717 16:01:33.995387 92606 docker.go:642] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0717 16:01:33.995437 92606 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0717 16:01:34.005190 92606 ssh_runner.go:195] Run: which lz4
I0717 16:01:34.009853 92606 ssh_runner.go:195] Run: stat -c "%s %y" /preloaded.tar.lz4
I0717 16:01:34.015163 92606 ssh_runner.go:352] existence check for /preloaded.tar.lz4: stat -c "%s %y" /preloaded.tar.lz4: Process exited with status 1
stdout:
stderr:
stat: cannot statx '/preloaded.tar.lz4': No such file or directory
I0717 16:01:34.015225 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 --> /preloaded.tar.lz4 (369789069 bytes)
I0717 16:01:39.803332 92606 docker.go:600] Took 5.793350 seconds to copy over tarball
I0717 16:01:39.803436 92606 ssh_runner.go:195] Run: sudo tar -I lz4 -C /var -xf /preloaded.tar.lz4
I0717 16:01:42.394194 92606 ssh_runner.go:235] Completed: sudo tar -I lz4 -C /var -xf /preloaded.tar.lz4: (2.590665838s)
I0717 16:01:42.394208 92606 ssh_runner.go:146] rm: /preloaded.tar.lz4
I0717 16:01:42.451968 92606 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0717 16:01:42.462476 92606 ssh_runner.go:362] scp memory --> /var/lib/docker/image/overlay2/repositories.json (2499 bytes)
I0717 16:01:42.481568 92606 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0717 16:01:42.563842 92606 ssh_runner.go:195] Run: sudo systemctl restart docker
I0717 16:01:43.046539 92606 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0717 16:01:43.069150 92606 docker.go:636] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0717 16:01:43.069169 92606 docker.go:642] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0717 16:01:43.069177 92606 cache_images.go:88] LoadImages start: [registry.k8s.io/kube-apiserver:v1.16.0 registry.k8s.io/kube-controller-manager:v1.16.0 registry.k8s.io/kube-scheduler:v1.16.0 registry.k8s.io/kube-proxy:v1.16.0 registry.k8s.io/pause:3.1 registry.k8s.io/etcd:3.3.15-0 registry.k8s.io/coredns:1.6.2 gcr.io/k8s-minikube/storage-provisioner:v5]
I0717 16:01:43.074793 92606 image.go:134] retrieving image: gcr.io/k8s-minikube/storage-provisioner:v5
I0717 16:01:43.074793 92606 image.go:134] retrieving image: registry.k8s.io/pause:3.1
I0717 16:01:43.075299 92606 image.go:134] retrieving image: registry.k8s.io/kube-apiserver:v1.16.0
I0717 16:01:43.075336 92606 image.go:134] retrieving image: registry.k8s.io/coredns:1.6.2
I0717 16:01:43.075378 92606 image.go:134] retrieving image: registry.k8s.io/kube-proxy:v1.16.0
I0717 16:01:43.075431 92606 image.go:134] retrieving image: registry.k8s.io/kube-scheduler:v1.16.0
I0717 16:01:43.076199 92606 image.go:134] retrieving image: registry.k8s.io/etcd:3.3.15-0
I0717 16:01:43.076279 92606 image.go:134] retrieving image: registry.k8s.io/kube-controller-manager:v1.16.0
I0717 16:01:43.080802 92606 image.go:177] daemon lookup for registry.k8s.io/kube-proxy:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-proxy:v1.16.0
I0717 16:01:43.081245 92606 image.go:177] daemon lookup for registry.k8s.io/etcd:3.3.15-0: Error response from daemon: No such image: registry.k8s.io/etcd:3.3.15-0
I0717 16:01:43.082081 92606 image.go:177] daemon lookup for registry.k8s.io/pause:3.1: Error response from daemon: No such image: registry.k8s.io/pause:3.1
I0717 16:01:43.082358 92606 image.go:177] daemon lookup for gcr.io/k8s-minikube/storage-provisioner:v5: Error response from daemon: No such image: gcr.io/k8s-minikube/storage-provisioner:v5
I0717 16:01:43.082550 92606 image.go:177] daemon lookup for registry.k8s.io/coredns:1.6.2: Error response from daemon: No such image: registry.k8s.io/coredns:1.6.2
I0717 16:01:43.082577 92606 image.go:177] daemon lookup for registry.k8s.io/kube-apiserver:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-apiserver:v1.16.0
I0717 16:01:43.082735 92606 image.go:177] daemon lookup for registry.k8s.io/kube-scheduler:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-scheduler:v1.16.0
I0717 16:01:43.084337 92606 image.go:177] daemon lookup for registry.k8s.io/kube-controller-manager:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-controller-manager:v1.16.0
I0717 16:01:44.308300 92606 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/etcd:3.3.15-0
I0717 16:01:44.331211 92606 cache_images.go:116] "registry.k8s.io/etcd:3.3.15-0" needs transfer: "registry.k8s.io/etcd:3.3.15-0" does not exist at hash "b2756210eeabf84f3221da9959e9483f3919dc2aaab4cd45e7cd072fcbde27ed" in container runtime
I0717 16:01:44.331248 92606 docker.go:316] Removing image: registry.k8s.io/etcd:3.3.15-0
I0717 16:01:44.331314 92606 ssh_runner.go:195] Run: docker rmi registry.k8s.io/etcd:3.3.15-0
I0717 16:01:44.352260 92606 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/16899-76867/.minikube/cache/images/amd64/registry.k8s.io/etcd_3.3.15-0
I0717 16:01:44.455104 92606 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-proxy:v1.16.0
I0717 16:01:44.477932 92606 cache_images.go:116] "registry.k8s.io/kube-proxy:v1.16.0" needs transfer: "registry.k8s.io/kube-proxy:v1.16.0" does not exist at hash "c21b0c7400f988db4777858edd13b6d3930d62d7ccf026d2415485a52037f384" in container runtime
I0717 16:01:44.477966 92606 docker.go:316] Removing image: registry.k8s.io/kube-proxy:v1.16.0
I0717 16:01:44.478028 92606 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-proxy:v1.16.0
I0717 16:01:44.501402 92606 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/16899-76867/.minikube/cache/images/amd64/registry.k8s.io/kube-proxy_v1.16.0
I0717 16:01:44.588611 92606 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/pause:3.1
I0717 16:01:44.613005 92606 cache_images.go:116] "registry.k8s.io/pause:3.1" needs transfer: "registry.k8s.io/pause:3.1" does not exist at hash "da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e" in container runtime
I0717 16:01:44.613038 92606 docker.go:316] Removing image: registry.k8s.io/pause:3.1
I0717 16:01:44.613092 92606 ssh_runner.go:195] Run: docker rmi registry.k8s.io/pause:3.1
I0717 16:01:44.636011 92606 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/16899-76867/.minikube/cache/images/amd64/registry.k8s.io/pause_3.1
I0717 16:01:45.109346 92606 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/coredns:1.6.2
I0717 16:01:45.132409 92606 cache_images.go:116] "registry.k8s.io/coredns:1.6.2" needs transfer: "registry.k8s.io/coredns:1.6.2" does not exist at hash "bf261d157914477ee1a5969d28ec687f3fbfc9fb5a664b22df78e57023b0e03b" in container runtime
I0717 16:01:45.132444 92606 docker.go:316] Removing image: registry.k8s.io/coredns:1.6.2
I0717 16:01:45.132509 92606 ssh_runner.go:195] Run: docker rmi registry.k8s.io/coredns:1.6.2
I0717 16:01:45.154147 92606 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/16899-76867/.minikube/cache/images/amd64/registry.k8s.io/coredns_1.6.2
I0717 16:01:45.421993 92606 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} gcr.io/k8s-minikube/storage-provisioner:v5
I0717 16:01:45.424304 92606 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-apiserver:v1.16.0
I0717 16:01:45.455206 92606 cache_images.go:116] "registry.k8s.io/kube-apiserver:v1.16.0" needs transfer: "registry.k8s.io/kube-apiserver:v1.16.0" does not exist at hash "b305571ca60a5a7818bda47da122683d75e8a1907475681ee8b1efbd06bff12e" in container runtime
I0717 16:01:45.455243 92606 docker.go:316] Removing image: registry.k8s.io/kube-apiserver:v1.16.0
I0717 16:01:45.455316 92606 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-apiserver:v1.16.0
I0717 16:01:45.479191 92606 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/16899-76867/.minikube/cache/images/amd64/registry.k8s.io/kube-apiserver_v1.16.0
I0717 16:01:45.755079 92606 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-scheduler:v1.16.0
I0717 16:01:45.777515 92606 cache_images.go:116] "registry.k8s.io/kube-scheduler:v1.16.0" needs transfer: "registry.k8s.io/kube-scheduler:v1.16.0" does not exist at hash "301ddc62b80b16315d3c2653cf3888370394277afb3187614cfa20edc352ca0a" in container runtime
I0717 16:01:45.777558 92606 docker.go:316] Removing image: registry.k8s.io/kube-scheduler:v1.16.0
I0717 16:01:45.777630 92606 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-scheduler:v1.16.0
I0717 16:01:45.801332 92606 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/16899-76867/.minikube/cache/images/amd64/registry.k8s.io/kube-scheduler_v1.16.0
I0717 16:01:46.350607 92606 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-controller-manager:v1.16.0
I0717 16:01:46.370043 92606 cache_images.go:116] "registry.k8s.io/kube-controller-manager:v1.16.0" needs transfer: "registry.k8s.io/kube-controller-manager:v1.16.0" does not exist at hash "06a629a7e51cdcc81a5ed6a3e6650348312f20c954ac52ee489a023628ec9c7d" in container runtime
I0717 16:01:46.370076 92606 docker.go:316] Removing image: registry.k8s.io/kube-controller-manager:v1.16.0
I0717 16:01:46.370148 92606 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-controller-manager:v1.16.0
I0717 16:01:46.390027 92606 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/16899-76867/.minikube/cache/images/amd64/registry.k8s.io/kube-controller-manager_v1.16.0
I0717 16:01:46.390073 92606 cache_images.go:92] LoadImages completed in 3.320798361s
W0717 16:01:46.390123 92606 out.go:239] X Unable to load cached images: loading cached images: stat /Users/jenkins/minikube-integration/16899-76867/.minikube/cache/images/amd64/registry.k8s.io/etcd_3.3.15-0: no such file or directory
X Unable to load cached images: loading cached images: stat /Users/jenkins/minikube-integration/16899-76867/.minikube/cache/images/amd64/registry.k8s.io/etcd_3.3.15-0: no such file or directory
I0717 16:01:46.390195 92606 ssh_runner.go:195] Run: docker info --format {{.CgroupDriver}}
I0717 16:01:46.446838 92606 cni.go:84] Creating CNI manager for ""
I0717 16:01:46.446855 92606 cni.go:168] CNI unnecessary in this configuration, recommending no CNI
I0717 16:01:46.446872 92606 kubeadm.go:87] Using pod CIDR: 10.244.0.0/16
I0717 16:01:46.446895 92606 kubeadm.go:176] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.76.2 APIServerPort:8443 KubernetesVersion:v1.16.0 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:old-k8s-version-770000 NodeName:old-k8s-version-770000 DNSDomain:cluster.local CRISocket:/var/run/dockershim.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.76.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.76.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticP
odPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[hairpinMode:hairpin-veth runtimeRequestTimeout:15m] PrependCriSocketUnix:false}
I0717 16:01:46.447001 92606 kubeadm.go:181] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.76.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: "old-k8s-version-770000"
kubeletExtraArgs:
node-ip: 192.168.76.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.76.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: old-k8s-version-770000
controlPlaneEndpoint: control-plane.minikube.internal:8443
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
listen-metrics-urls: http://127.0.0.1:2381,http://192.168.76.2:2381
kubernetesVersion: v1.16.0
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0717 16:01:46.447070 92606 kubeadm.go:976] kubelet [Unit]
Wants=docker.socket
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.16.0/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=old-k8s-version-770000 --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.76.2
[Install]
config:
{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-770000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:}
I0717 16:01:46.447135 92606 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.16.0
I0717 16:01:46.457801 92606 binaries.go:44] Found k8s binaries, skipping transfer
I0717 16:01:46.457862 92606 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0717 16:01:46.469148 92606 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (348 bytes)
I0717 16:01:46.488380 92606 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0717 16:01:46.507522 92606 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2174 bytes)
I0717 16:01:46.528271 92606 ssh_runner.go:195] Run: grep 192.168.76.2 control-plane.minikube.internal$ /etc/hosts
I0717 16:01:46.533254 92606 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.76.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0717 16:01:46.546751 92606 certs.go:56] Setting up /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000 for IP: 192.168.76.2
I0717 16:01:46.546784 92606 certs.go:190] acquiring lock for shared ca certs: {Name:mk8dc1f2afa352f9c2168154d4ab47beda1b6a37 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0717 16:01:46.546989 92606 certs.go:199] skipping minikubeCA CA generation: /Users/jenkins/minikube-integration/16899-76867/.minikube/ca.key
I0717 16:01:46.547084 92606 certs.go:199] skipping proxyClientCA CA generation: /Users/jenkins/minikube-integration/16899-76867/.minikube/proxy-client-ca.key
I0717 16:01:46.547154 92606 certs.go:319] generating minikube-user signed cert: /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/client.key
I0717 16:01:46.547175 92606 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/client.crt with IP's: []
I0717 16:01:46.662246 92606 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/client.crt ...
I0717 16:01:46.662270 92606 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/client.crt: {Name:mk381354864b59b07e58991d1d9d8a70c4426bab Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0717 16:01:46.662712 92606 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/client.key ...
I0717 16:01:46.662722 92606 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/client.key: {Name:mkaf6c5fc183fa44b46c75514c440f7970af7573 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0717 16:01:46.662978 92606 certs.go:319] generating minikube signed cert: /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/apiserver.key.31bdca25
I0717 16:01:46.663012 92606 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/apiserver.crt.31bdca25 with IP's: [192.168.76.2 10.96.0.1 127.0.0.1 10.0.0.1]
I0717 16:01:46.736349 92606 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/apiserver.crt.31bdca25 ...
I0717 16:01:46.736367 92606 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/apiserver.crt.31bdca25: {Name:mk3968f378081303c69947e75cea3b03adaa7ba7 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0717 16:01:46.736691 92606 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/apiserver.key.31bdca25 ...
I0717 16:01:46.736700 92606 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/apiserver.key.31bdca25: {Name:mk810bdeb5511fa3db2c0c958a7bffafc4aff436 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0717 16:01:46.736915 92606 certs.go:337] copying /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/apiserver.crt.31bdca25 -> /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/apiserver.crt
I0717 16:01:46.737115 92606 certs.go:341] copying /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/apiserver.key.31bdca25 -> /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/apiserver.key
I0717 16:01:46.737287 92606 certs.go:319] generating aggregator signed cert: /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/proxy-client.key
I0717 16:01:46.737354 92606 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/proxy-client.crt with IP's: []
I0717 16:01:46.781045 92606 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/proxy-client.crt ...
I0717 16:01:46.781061 92606 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/proxy-client.crt: {Name:mkfddc52176926356badd66cf60b995a8c93394d Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0717 16:01:46.781390 92606 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/proxy-client.key ...
I0717 16:01:46.781399 92606 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/proxy-client.key: {Name:mk8267b910eec7822db3634557d1804094a71d0a Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0717 16:01:46.781833 92606 certs.go:437] found cert: /Users/jenkins/minikube-integration/16899-76867/.minikube/certs/Users/jenkins/minikube-integration/16899-76867/.minikube/certs/77324.pem (1338 bytes)
W0717 16:01:46.781888 92606 certs.go:433] ignoring /Users/jenkins/minikube-integration/16899-76867/.minikube/certs/Users/jenkins/minikube-integration/16899-76867/.minikube/certs/77324_empty.pem, impossibly tiny 0 bytes
I0717 16:01:46.781905 92606 certs.go:437] found cert: /Users/jenkins/minikube-integration/16899-76867/.minikube/certs/Users/jenkins/minikube-integration/16899-76867/.minikube/certs/ca-key.pem (1679 bytes)
I0717 16:01:46.781941 92606 certs.go:437] found cert: /Users/jenkins/minikube-integration/16899-76867/.minikube/certs/Users/jenkins/minikube-integration/16899-76867/.minikube/certs/ca.pem (1078 bytes)
I0717 16:01:46.781973 92606 certs.go:437] found cert: /Users/jenkins/minikube-integration/16899-76867/.minikube/certs/Users/jenkins/minikube-integration/16899-76867/.minikube/certs/cert.pem (1123 bytes)
I0717 16:01:46.782008 92606 certs.go:437] found cert: /Users/jenkins/minikube-integration/16899-76867/.minikube/certs/Users/jenkins/minikube-integration/16899-76867/.minikube/certs/key.pem (1675 bytes)
I0717 16:01:46.782082 92606 certs.go:437] found cert: /Users/jenkins/minikube-integration/16899-76867/.minikube/files/etc/ssl/certs/Users/jenkins/minikube-integration/16899-76867/.minikube/files/etc/ssl/certs/773242.pem (1708 bytes)
I0717 16:01:46.782657 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1399 bytes)
I0717 16:01:46.805346 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1679 bytes)
I0717 16:01:46.829912 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0717 16:01:46.854243 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/profiles/old-k8s-version-770000/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1675 bytes)
I0717 16:01:46.878203 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0717 16:01:46.903705 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1679 bytes)
I0717 16:01:46.928736 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0717 16:01:46.955293 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1679 bytes)
I0717 16:01:46.980214 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0717 16:01:47.005053 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/certs/77324.pem --> /usr/share/ca-certificates/77324.pem (1338 bytes)
I0717 16:01:47.031458 92606 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/16899-76867/.minikube/files/etc/ssl/certs/773242.pem --> /usr/share/ca-certificates/773242.pem (1708 bytes)
I0717 16:01:47.058211 92606 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0717 16:01:47.077744 92606 ssh_runner.go:195] Run: openssl version
I0717 16:01:47.084658 92606 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/77324.pem && ln -fs /usr/share/ca-certificates/77324.pem /etc/ssl/certs/77324.pem"
I0717 16:01:47.095943 92606 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/77324.pem
I0717 16:01:47.101644 92606 certs.go:480] hashing: -rw-r--r-- 1 root root 1338 Jul 17 22:13 /usr/share/ca-certificates/77324.pem
I0717 16:01:47.101713 92606 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/77324.pem
I0717 16:01:47.110161 92606 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/77324.pem /etc/ssl/certs/51391683.0"
I0717 16:01:47.121518 92606 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/773242.pem && ln -fs /usr/share/ca-certificates/773242.pem /etc/ssl/certs/773242.pem"
I0717 16:01:47.133664 92606 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/773242.pem
I0717 16:01:47.138463 92606 certs.go:480] hashing: -rw-r--r-- 1 root root 1708 Jul 17 22:13 /usr/share/ca-certificates/773242.pem
I0717 16:01:47.138524 92606 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/773242.pem
I0717 16:01:47.145895 92606 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/773242.pem /etc/ssl/certs/3ec20f2e.0"
I0717 16:01:47.156109 92606 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0717 16:01:47.166062 92606 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0717 16:01:47.170876 92606 certs.go:480] hashing: -rw-r--r-- 1 root root 1111 Jul 17 22:07 /usr/share/ca-certificates/minikubeCA.pem
I0717 16:01:47.170928 92606 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0717 16:01:47.178212 92606 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0717 16:01:47.187971 92606 ssh_runner.go:195] Run: ls /var/lib/minikube/certs/etcd
I0717 16:01:47.192545 92606 certs.go:353] certs directory doesn't exist, likely first start: ls /var/lib/minikube/certs/etcd: Process exited with status 2
stdout:
stderr:
ls: cannot access '/var/lib/minikube/certs/etcd': No such file or directory
I0717 16:01:47.192591 92606 kubeadm.go:404] StartCluster: {Name:old-k8s-version-770000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-770000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs
:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQe
muFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0}
I0717 16:01:47.192699 92606 ssh_runner.go:195] Run: docker ps --filter status=paused --filter=name=k8s_.*_(kube-system)_ --format={{.ID}}
I0717 16:01:47.213075 92606 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0717 16:01:47.224484 92606 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0717 16:01:47.240494 92606 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0717 16:01:47.240567 92606 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0717 16:01:47.250845 92606 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0717 16:01:47.250880 92606 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0717 16:01:47.311048 92606 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0717 16:01:47.311270 92606 kubeadm.go:322] [preflight] Running pre-flight checks
I0717 16:01:47.595202 92606 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0717 16:01:47.595313 92606 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0717 16:01:47.595393 92606 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0717 16:01:47.796129 92606 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0717 16:01:47.796978 92606 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0717 16:01:47.803893 92606 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0717 16:01:47.887163 92606 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0717 16:01:47.908783 92606 out.go:204] - Generating certificates and keys ...
I0717 16:01:47.908873 92606 kubeadm.go:322] [certs] Using existing ca certificate authority
I0717 16:01:47.908961 92606 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0717 16:01:48.107400 92606 kubeadm.go:322] [certs] Generating "apiserver-kubelet-client" certificate and key
I0717 16:01:48.172101 92606 kubeadm.go:322] [certs] Generating "front-proxy-ca" certificate and key
I0717 16:01:48.352959 92606 kubeadm.go:322] [certs] Generating "front-proxy-client" certificate and key
I0717 16:01:48.533240 92606 kubeadm.go:322] [certs] Generating "etcd/ca" certificate and key
I0717 16:01:48.623978 92606 kubeadm.go:322] [certs] Generating "etcd/server" certificate and key
I0717 16:01:48.624092 92606 kubeadm.go:322] [certs] etcd/server serving cert is signed for DNS names [old-k8s-version-770000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
I0717 16:01:48.701963 92606 kubeadm.go:322] [certs] Generating "etcd/peer" certificate and key
I0717 16:01:48.702139 92606 kubeadm.go:322] [certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-770000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
I0717 16:01:48.989598 92606 kubeadm.go:322] [certs] Generating "etcd/healthcheck-client" certificate and key
I0717 16:01:49.136647 92606 kubeadm.go:322] [certs] Generating "apiserver-etcd-client" certificate and key
I0717 16:01:49.216225 92606 kubeadm.go:322] [certs] Generating "sa" key and public key
I0717 16:01:49.216427 92606 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0717 16:01:49.415632 92606 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0717 16:01:49.645563 92606 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0717 16:01:49.861489 92606 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0717 16:01:50.022126 92606 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0717 16:01:50.022826 92606 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0717 16:01:50.064539 92606 out.go:204] - Booting up control plane ...
I0717 16:01:50.064696 92606 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0717 16:01:50.064792 92606 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0717 16:01:50.064906 92606 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0717 16:01:50.065020 92606 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0717 16:01:50.065213 92606 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0717 16:02:30.034609 92606 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0717 16:02:30.036011 92606 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0717 16:02:30.036272 92606 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0717 16:02:35.036474 92606 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0717 16:02:35.036648 92606 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0717 16:02:45.037397 92606 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0717 16:02:45.037558 92606 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0717 16:03:05.038653 92606 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0717 16:03:05.038815 92606 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0717 16:03:45.040508 92606 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0717 16:03:45.040718 92606 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0717 16:03:45.040727 92606 kubeadm.go:322]
I0717 16:03:45.040755 92606 kubeadm.go:322] Unfortunately, an error has occurred:
I0717 16:03:45.040799 92606 kubeadm.go:322] timed out waiting for the condition
I0717 16:03:45.040826 92606 kubeadm.go:322]
I0717 16:03:45.040869 92606 kubeadm.go:322] This error is likely caused by:
I0717 16:03:45.040897 92606 kubeadm.go:322] - The kubelet is not running
I0717 16:03:45.041013 92606 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0717 16:03:45.041028 92606 kubeadm.go:322]
I0717 16:03:45.041153 92606 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0717 16:03:45.041204 92606 kubeadm.go:322] - 'systemctl status kubelet'
I0717 16:03:45.041246 92606 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0717 16:03:45.041261 92606 kubeadm.go:322]
I0717 16:03:45.041383 92606 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0717 16:03:45.041497 92606 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0717 16:03:45.041641 92606 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0717 16:03:45.041719 92606 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0717 16:03:45.041833 92606 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0717 16:03:45.041936 92606 kubeadm.go:322] - 'docker logs CONTAINERID'
I0717 16:03:45.044408 92606 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0717 16:03:45.044511 92606 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0717 16:03:45.044698 92606 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.4. Latest validated version: 18.09
I0717 16:03:45.044810 92606 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0717 16:03:45.044972 92606 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0717 16:03:45.045105 92606 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
W0717 16:03:45.045253 92606 out.go:239] ! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-770000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-770000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.4. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-770000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-770000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.4. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
I0717 16:03:45.045315 92606 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm reset --cri-socket /var/run/dockershim.sock --force"
I0717 16:03:45.485166 92606 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service kubelet
I0717 16:03:45.497894 92606 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0717 16:03:45.497963 92606 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0717 16:03:45.508170 92606 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0717 16:03:45.508194 92606 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0717 16:03:45.682722 92606 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0717 16:03:45.682850 92606 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0717 16:03:45.739572 92606 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.4. Latest validated version: 18.09
I0717 16:03:45.819796 92606 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0717 16:05:41.553902 92606 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0717 16:05:41.553982 92606 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
I0717 16:05:41.557090 92606 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0717 16:05:41.557137 92606 kubeadm.go:322] [preflight] Running pre-flight checks
I0717 16:05:41.557196 92606 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0717 16:05:41.557278 92606 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0717 16:05:41.557363 92606 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0717 16:05:41.557460 92606 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0717 16:05:41.557531 92606 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0717 16:05:41.557580 92606 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0717 16:05:41.557634 92606 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0717 16:05:41.579159 92606 out.go:204] - Generating certificates and keys ...
I0717 16:05:41.579266 92606 kubeadm.go:322] [certs] Using existing ca certificate authority
I0717 16:05:41.579407 92606 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0717 16:05:41.579540 92606 kubeadm.go:322] [certs] Using existing apiserver-kubelet-client certificate and key on disk
I0717 16:05:41.579672 92606 kubeadm.go:322] [certs] Using existing front-proxy-ca certificate authority
I0717 16:05:41.579805 92606 kubeadm.go:322] [certs] Using existing front-proxy-client certificate and key on disk
I0717 16:05:41.579893 92606 kubeadm.go:322] [certs] Using existing etcd/ca certificate authority
I0717 16:05:41.580003 92606 kubeadm.go:322] [certs] Using existing etcd/server certificate and key on disk
I0717 16:05:41.580097 92606 kubeadm.go:322] [certs] Using existing etcd/peer certificate and key on disk
I0717 16:05:41.580204 92606 kubeadm.go:322] [certs] Using existing etcd/healthcheck-client certificate and key on disk
I0717 16:05:41.580305 92606 kubeadm.go:322] [certs] Using existing apiserver-etcd-client certificate and key on disk
I0717 16:05:41.580362 92606 kubeadm.go:322] [certs] Using the existing "sa" key
I0717 16:05:41.580448 92606 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0717 16:05:41.580522 92606 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0717 16:05:41.580622 92606 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0717 16:05:41.580737 92606 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0717 16:05:41.580826 92606 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0717 16:05:41.580948 92606 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0717 16:05:41.641928 92606 out.go:204] - Booting up control plane ...
I0717 16:05:41.642105 92606 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0717 16:05:41.642241 92606 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0717 16:05:41.642349 92606 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0717 16:05:41.642491 92606 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0717 16:05:41.642759 92606 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0717 16:05:41.642850 92606 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0717 16:05:41.642964 92606 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0717 16:05:41.643302 92606 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0717 16:05:41.643431 92606 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0717 16:05:41.643716 92606 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0717 16:05:41.643850 92606 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0717 16:05:41.644149 92606 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0717 16:05:41.644265 92606 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0717 16:05:41.644548 92606 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0717 16:05:41.644669 92606 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0717 16:05:41.644871 92606 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0717 16:05:41.644885 92606 kubeadm.go:322]
I0717 16:05:41.644925 92606 kubeadm.go:322] Unfortunately, an error has occurred:
I0717 16:05:41.644973 92606 kubeadm.go:322] timed out waiting for the condition
I0717 16:05:41.644984 92606 kubeadm.go:322]
I0717 16:05:41.645022 92606 kubeadm.go:322] This error is likely caused by:
I0717 16:05:41.645056 92606 kubeadm.go:322] - The kubelet is not running
I0717 16:05:41.645165 92606 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0717 16:05:41.645174 92606 kubeadm.go:322]
I0717 16:05:41.645291 92606 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0717 16:05:41.645330 92606 kubeadm.go:322] - 'systemctl status kubelet'
I0717 16:05:41.645365 92606 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0717 16:05:41.645373 92606 kubeadm.go:322]
I0717 16:05:41.645489 92606 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0717 16:05:41.645586 92606 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0717 16:05:41.645688 92606 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0717 16:05:41.645742 92606 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0717 16:05:41.645831 92606 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0717 16:05:41.645883 92606 kubeadm.go:322] - 'docker logs CONTAINERID'
I0717 16:05:41.645911 92606 kubeadm.go:406] StartCluster complete in 3m54.450422661s
I0717 16:05:41.646034 92606 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-apiserver --format={{.ID}}
I0717 16:05:41.665901 92606 logs.go:284] 0 containers: []
W0717 16:05:41.665915 92606 logs.go:286] No container was found matching "kube-apiserver"
I0717 16:05:41.665983 92606 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_etcd --format={{.ID}}
I0717 16:05:41.685039 92606 logs.go:284] 0 containers: []
W0717 16:05:41.685053 92606 logs.go:286] No container was found matching "etcd"
I0717 16:05:41.685123 92606 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_coredns --format={{.ID}}
I0717 16:05:41.705372 92606 logs.go:284] 0 containers: []
W0717 16:05:41.705384 92606 logs.go:286] No container was found matching "coredns"
I0717 16:05:41.705458 92606 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-scheduler --format={{.ID}}
I0717 16:05:41.725851 92606 logs.go:284] 0 containers: []
W0717 16:05:41.725875 92606 logs.go:286] No container was found matching "kube-scheduler"
I0717 16:05:41.725945 92606 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-proxy --format={{.ID}}
I0717 16:05:41.746212 92606 logs.go:284] 0 containers: []
W0717 16:05:41.746228 92606 logs.go:286] No container was found matching "kube-proxy"
I0717 16:05:41.746304 92606 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-controller-manager --format={{.ID}}
I0717 16:05:41.766612 92606 logs.go:284] 0 containers: []
W0717 16:05:41.766630 92606 logs.go:286] No container was found matching "kube-controller-manager"
I0717 16:05:41.766708 92606 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kindnet --format={{.ID}}
I0717 16:05:41.789636 92606 logs.go:284] 0 containers: []
W0717 16:05:41.789654 92606 logs.go:286] No container was found matching "kindnet"
I0717 16:05:41.789668 92606 logs.go:123] Gathering logs for container status ...
I0717 16:05:41.789679 92606 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0717 16:05:41.845301 92606 logs.go:123] Gathering logs for kubelet ...
I0717 16:05:41.845322 92606 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
I0717 16:05:41.885248 92606 logs.go:123] Gathering logs for dmesg ...
I0717 16:05:41.885262 92606 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0717 16:05:41.899988 92606 logs.go:123] Gathering logs for describe nodes ...
I0717 16:05:41.900003 92606 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0717 16:05:41.955719 92606 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0717 16:05:41.955735 92606 logs.go:123] Gathering logs for Docker ...
I0717 16:05:41.955741 92606 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u docker -u cri-docker -n 400"
W0717 16:05:41.972304 92606 out.go:369] Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.4. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0717 16:05:41.972324 92606 out.go:239] *
*
W0717 16:05:41.972363 92606 out.go:239] X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.4. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.4. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0717 16:05:41.972375 92606 out.go:239] *
*
W0717 16:05:41.972998 92606 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0717 16:05:42.057804 92606 out.go:177]
W0717 16:05:42.099686 92606 out.go:239] X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.4. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.4. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0717 16:05:42.099765 92606 out.go:239] * Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
* Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
W0717 16:05:42.099797 92606 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/4172
* Related issue: https://github.com/kubernetes/minikube/issues/4172
I0717 16:05:42.121903 92606 out.go:177]
** /stderr **
start_stop_delete_test.go:188: failed starting minikube -first start-. args "out/minikube-darwin-amd64 start -p old-k8s-version-770000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0": exit status 109
helpers_test.go:222: -----------------------post-mortem--------------------------------
helpers_test.go:230: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/FirstStart]: docker inspect <======
helpers_test.go:231: (dbg) Run: docker inspect old-k8s-version-770000
helpers_test.go:235: (dbg) docker inspect old-k8s-version-770000:
-- stdout --
[
{
"Id": "6129a8b881bae9c9b14658d603684185fa98d048ad62c6ede03346a49e6e2b56",
"Created": "2023-07-17T23:01:29.298658175Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1218635,
"ExitCode": 0,
"Error": "",
"StartedAt": "2023-07-17T23:01:29.516403393Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:c6cc01e6091959400f260dc442708e7c71630b58dab1f7c344cb00926bd84950",
"ResolvConfPath": "/var/lib/docker/containers/6129a8b881bae9c9b14658d603684185fa98d048ad62c6ede03346a49e6e2b56/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/6129a8b881bae9c9b14658d603684185fa98d048ad62c6ede03346a49e6e2b56/hostname",
"HostsPath": "/var/lib/docker/containers/6129a8b881bae9c9b14658d603684185fa98d048ad62c6ede03346a49e6e2b56/hosts",
"LogPath": "/var/lib/docker/containers/6129a8b881bae9c9b14658d603684185fa98d048ad62c6ede03346a49e6e2b56/6129a8b881bae9c9b14658d603684185fa98d048ad62c6ede03346a49e6e2b56-json.log",
"Name": "/old-k8s-version-770000",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"old-k8s-version-770000:/var",
"/lib/modules:/lib/modules:ro"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "old-k8s-version-770000",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 2306867200,
"NanoCpus": 2000000000,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": [],
"BlkioDeviceWriteBps": [],
"BlkioDeviceReadIOps": [],
"BlkioDeviceWriteIOps": [],
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 2306867200,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/a038e5269af2118ee927b0485208b6f3b1d1f1a742907462c43ed3f30ca09e24-init/diff:/var/lib/docker/overlay2/388817d1807139a2b5fe2987f16fc65d58f6720a0b0343097a59eb837a278a0e/diff",
"MergedDir": "/var/lib/docker/overlay2/a038e5269af2118ee927b0485208b6f3b1d1f1a742907462c43ed3f30ca09e24/merged",
"UpperDir": "/var/lib/docker/overlay2/a038e5269af2118ee927b0485208b6f3b1d1f1a742907462c43ed3f30ca09e24/diff",
"WorkDir": "/var/lib/docker/overlay2/a038e5269af2118ee927b0485208b6f3b1d1f1a742907462c43ed3f30ca09e24/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "volume",
"Name": "old-k8s-version-770000",
"Source": "/var/lib/docker/volumes/old-k8s-version-770000/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
},
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
}
],
"Config": {
"Hostname": "old-k8s-version-770000",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"22/tcp": {},
"2376/tcp": {},
"32443/tcp": {},
"5000/tcp": {},
"8443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase:v0.0.40@sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "old-k8s-version-770000",
"name.minikube.sigs.k8s.io": "old-k8s-version-770000",
"role.minikube.sigs.k8s.io": ""
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "c07258dac6be990577a4e3f1e6cbd9a4759194f33d0f96dce83e5e8558aeddb0",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "57123"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "57124"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "57125"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "57126"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "57127"
}
]
},
"SandboxKey": "/var/run/docker/netns/c07258dac6be",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"old-k8s-version-770000": {
"IPAMConfig": {
"IPv4Address": "192.168.76.2"
},
"Links": null,
"Aliases": [
"6129a8b881ba",
"old-k8s-version-770000"
],
"NetworkID": "e0b81b03df244d0caf05aedc1b790fca29cd02fdbba810fc90a219bab32afcb3",
"EndpointID": "4fd6bd6d37711f3e0f856445ac094c78f36caf611e81b86031855dfec1573cf4",
"Gateway": "192.168.76.1",
"IPAddress": "192.168.76.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:c0:a8:4c:02",
"DriverOpts": null
}
}
}
}
]
-- /stdout --
helpers_test.go:239: (dbg) Run: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-770000 -n old-k8s-version-770000
helpers_test.go:239: (dbg) Non-zero exit: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-770000 -n old-k8s-version-770000: exit status 6 (360.591979ms)
-- stdout --
Running
WARNING: Your kubectl is pointing to stale minikube-vm.
To fix the kubectl context, run `minikube update-context`
-- /stdout --
** stderr **
E0717 16:05:42.635115 93736 status.go:415] kubeconfig endpoint: extract IP: "old-k8s-version-770000" does not appear in /Users/jenkins/minikube-integration/16899-76867/kubeconfig
** /stderr **
helpers_test.go:239: status error: exit status 6 (may be ok)
helpers_test.go:241: "old-k8s-version-770000" host is not running, skipping log retrieval (state="Running\nWARNING: Your kubectl is pointing to stale minikube-vm.\nTo fix the kubectl context, run `minikube update-context`")
--- FAIL: TestStartStop/group/old-k8s-version/serial/FirstStart (258.14s)