=== RUN TestStartStop/group/old-k8s-version/serial/FirstStart
start_stop_delete_test.go:186: (dbg) Run: out/minikube-darwin-amd64 start -p old-k8s-version-745000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0
E0213 15:56:56.117672 6776 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/custom-flannel-208000/client.crt: no such file or directory
E0213 15:57:06.357737 6776 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/custom-flannel-208000/client.crt: no such file or directory
E0213 15:57:09.209843 6776 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/calico-208000/client.crt: no such file or directory
start_stop_delete_test.go:186: (dbg) Non-zero exit: out/minikube-darwin-amd64 start -p old-k8s-version-745000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0: exit status 109 (4m19.180823318s)
-- stdout --
* [old-k8s-version-745000] minikube v1.32.0 on Darwin 14.3.1
- MINIKUBE_LOCATION=18169
- KUBECONFIG=/Users/jenkins/minikube-integration/18169-6320/kubeconfig
- MINIKUBE_BIN=out/minikube-darwin-amd64
- MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
- MINIKUBE_HOME=/Users/jenkins/minikube-integration/18169-6320/.minikube
- MINIKUBE_FORCE_SYSTEMD=
* Using the docker driver based on user configuration
* Using Docker Desktop driver with root privileges
* Starting control plane node old-k8s-version-745000 in cluster old-k8s-version-745000
* Pulling base image v0.0.42-1704759386-17866 ...
* Creating docker container (CPUs=2, Memory=2200MB) ...
* Preparing Kubernetes v1.16.0 on Docker 24.0.7 ...
- Generating certificates and keys ...
- Booting up control plane ...
- Generating certificates and keys ...
- Booting up control plane ...
-- /stdout --
** stderr **
I0213 15:56:51.498081 22261 out.go:291] Setting OutFile to fd 1 ...
I0213 15:56:51.498272 22261 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0213 15:56:51.498279 22261 out.go:304] Setting ErrFile to fd 2...
I0213 15:56:51.498283 22261 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0213 15:56:51.498534 22261 root.go:338] Updating PATH: /Users/jenkins/minikube-integration/18169-6320/.minikube/bin
I0213 15:56:51.500267 22261 out.go:298] Setting JSON to false
I0213 15:56:51.523503 22261 start.go:128] hostinfo: {"hostname":"MacOS-Agent-1.local","uptime":5471,"bootTime":1707863140,"procs":522,"os":"darwin","platform":"darwin","platformFamily":"Standalone Workstation","platformVersion":"14.3.1","kernelVersion":"23.3.0","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"b7610dcb-1435-5842-8d5a-b2388403fea3"}
W0213 15:56:51.523615 22261 start.go:136] gopshost.Virtualization returned error: not implemented yet
I0213 15:56:51.545670 22261 out.go:177] * [old-k8s-version-745000] minikube v1.32.0 on Darwin 14.3.1
I0213 15:56:51.588563 22261 out.go:177] - MINIKUBE_LOCATION=18169
I0213 15:56:51.588632 22261 notify.go:220] Checking for updates...
I0213 15:56:51.631418 22261 out.go:177] - KUBECONFIG=/Users/jenkins/minikube-integration/18169-6320/kubeconfig
I0213 15:56:51.652526 22261 out.go:177] - MINIKUBE_BIN=out/minikube-darwin-amd64
I0213 15:56:51.674368 22261 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0213 15:56:51.695394 22261 out.go:177] - MINIKUBE_HOME=/Users/jenkins/minikube-integration/18169-6320/.minikube
I0213 15:56:51.716357 22261 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0213 15:56:51.737960 22261 config.go:182] Loaded profile config "bridge-208000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.28.4
I0213 15:56:51.738138 22261 driver.go:392] Setting default libvirt URI to qemu:///system
I0213 15:56:51.795875 22261 docker.go:122] docker version: linux-25.0.3:Docker Desktop 4.27.2 (137060)
I0213 15:56:51.796044 22261 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0213 15:56:51.901668 22261 info.go:266] docker info: {ID:37b3081a-8cd6-457e-b2a4-79dc82345b06 Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:73 OomKillDisable:false NGoroutines:115 SystemTime:2024-02-13 23:56:51.891949316 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:26 KernelVersion:6.6.12-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServerAddress:
https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:12 MemTotal:6213292032 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=unconfined name
=cgroupns] ProductLicense: Warnings:[WARNING: daemon is not using the default seccomp profile] ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1-desktop.4] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.5-desktop.1] map[Name:debug Path:/Users/jenkins/.docker/cli-plugins/docker-debug SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-debug] ShortDescription:Get a shell into any image or container. Vendor:Docker Inc. Version:0.0.24] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker D
ev Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Manages Docker extensions Vendor:Docker Inc. Version:v0.2.21] map[Name:feedback Path:/Users/jenkins/.docker/cli-plugins/docker-feedback SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-feedback] ShortDescription:Provide feedback, right in your terminal! Vendor:Docker Inc. Version:v1.0.4] map[Name:init Path:/Users/jenkins/.docker/cli-plugins/docker-init SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-init] ShortDescription:Creates Docker-related starter files for your project Vendor:Docker Inc. Version:v1.0.0] map[Name:sbom Path:/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM)
for an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Err:failed to fetch metadata: fork/exec /Users/jenkins/.docker/cli-plugins/docker-scan: no such file or directory Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Docker Scout Vendor:Docker Inc. Version:v1.4.1]] Warnings:<nil>}}
I0213 15:56:51.944137 22261 out.go:177] * Using the docker driver based on user configuration
I0213 15:56:51.967030 22261 start.go:298] selected driver: docker
I0213 15:56:51.967045 22261 start.go:902] validating driver "docker" against <nil>
I0213 15:56:51.967055 22261 start.go:913] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0213 15:56:51.970316 22261 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0213 15:56:52.076663 22261 info.go:266] docker info: {ID:37b3081a-8cd6-457e-b2a4-79dc82345b06 Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:73 OomKillDisable:false NGoroutines:115 SystemTime:2024-02-13 23:56:52.065582055 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:26 KernelVersion:6.6.12-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServerAddress:
https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:12 MemTotal:6213292032 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=unconfined name
=cgroupns] ProductLicense: Warnings:[WARNING: daemon is not using the default seccomp profile] ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1-desktop.4] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.5-desktop.1] map[Name:debug Path:/Users/jenkins/.docker/cli-plugins/docker-debug SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-debug] ShortDescription:Get a shell into any image or container. Vendor:Docker Inc. Version:0.0.24] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker D
ev Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Manages Docker extensions Vendor:Docker Inc. Version:v0.2.21] map[Name:feedback Path:/Users/jenkins/.docker/cli-plugins/docker-feedback SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-feedback] ShortDescription:Provide feedback, right in your terminal! Vendor:Docker Inc. Version:v1.0.4] map[Name:init Path:/Users/jenkins/.docker/cli-plugins/docker-init SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-init] ShortDescription:Creates Docker-related starter files for your project Vendor:Docker Inc. Version:v1.0.0] map[Name:sbom Path:/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM)
for an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Err:failed to fetch metadata: fork/exec /Users/jenkins/.docker/cli-plugins/docker-scan: no such file or directory Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Docker Scout Vendor:Docker Inc. Version:v1.4.1]] Warnings:<nil>}}
I0213 15:56:52.077004 22261 start_flags.go:307] no existing cluster config was found, will generate one from the flags
I0213 15:56:52.077412 22261 start_flags.go:927] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0213 15:56:52.099178 22261 out.go:177] * Using Docker Desktop driver with root privileges
I0213 15:56:52.120919 22261 cni.go:84] Creating CNI manager for ""
I0213 15:56:52.120951 22261 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0213 15:56:52.120963 22261 start_flags.go:321] config:
{Name:old-k8s-version-745000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-745000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local Containe
rRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs:}
I0213 15:56:52.142646 22261 out.go:177] * Starting control plane node old-k8s-version-745000 in cluster old-k8s-version-745000
I0213 15:56:52.184889 22261 cache.go:121] Beginning downloading kic base image for docker with docker
I0213 15:56:52.226819 22261 out.go:177] * Pulling base image v0.0.42-1704759386-17866 ...
I0213 15:56:52.247786 22261 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0213 15:56:52.247805 22261 image.go:79] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 in local docker daemon
I0213 15:56:52.247824 22261 preload.go:148] Found local preload: /Users/jenkins/minikube-integration/18169-6320/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4
I0213 15:56:52.247841 22261 cache.go:56] Caching tarball of preloaded images
I0213 15:56:52.247941 22261 preload.go:174] Found /Users/jenkins/minikube-integration/18169-6320/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0213 15:56:52.247951 22261 cache.go:59] Finished verifying existence of preloaded tar for v1.16.0 on docker
I0213 15:56:52.248503 22261 profile.go:148] Saving config to /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/config.json ...
I0213 15:56:52.248598 22261 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/config.json: {Name:mk66ebef5bde6debcea5a2640297c9a4965b1ec3 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 15:56:52.298339 22261 image.go:83] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 in local docker daemon, skipping pull
I0213 15:56:52.298358 22261 cache.go:144] gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 exists in daemon, skipping load
I0213 15:56:52.298378 22261 cache.go:194] Successfully downloaded all kic artifacts
I0213 15:56:52.298430 22261 start.go:365] acquiring machines lock for old-k8s-version-745000: {Name:mkd7f9273d4ef06a0c4934b33030a9cfbc88fa9a Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0213 15:56:52.298589 22261 start.go:369] acquired machines lock for "old-k8s-version-745000" in 145.639µs
I0213 15:56:52.298617 22261 start.go:93] Provisioning new machine with config: &{Name:old-k8s-version-745000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-745000 Namespace:default APIServer
Name:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:f
alse DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs:} &{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}
I0213 15:56:52.298948 22261 start.go:125] createHost starting for "" (driver="docker")
I0213 15:56:52.320758 22261 out.go:204] * Creating docker container (CPUs=2, Memory=2200MB) ...
I0213 15:56:52.321220 22261 start.go:159] libmachine.API.Create for "old-k8s-version-745000" (driver="docker")
I0213 15:56:52.321272 22261 client.go:168] LocalClient.Create starting
I0213 15:56:52.321496 22261 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/18169-6320/.minikube/certs/ca.pem
I0213 15:56:52.321587 22261 main.go:141] libmachine: Decoding PEM data...
I0213 15:56:52.321613 22261 main.go:141] libmachine: Parsing certificate...
I0213 15:56:52.321697 22261 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/18169-6320/.minikube/certs/cert.pem
I0213 15:56:52.321754 22261 main.go:141] libmachine: Decoding PEM data...
I0213 15:56:52.321766 22261 main.go:141] libmachine: Parsing certificate...
I0213 15:56:52.342472 22261 cli_runner.go:164] Run: docker network inspect old-k8s-version-745000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
W0213 15:56:52.394660 22261 cli_runner.go:211] docker network inspect old-k8s-version-745000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1
I0213 15:56:52.394763 22261 network_create.go:281] running [docker network inspect old-k8s-version-745000] to gather additional debugging logs...
I0213 15:56:52.394780 22261 cli_runner.go:164] Run: docker network inspect old-k8s-version-745000
W0213 15:56:52.447572 22261 cli_runner.go:211] docker network inspect old-k8s-version-745000 returned with exit code 1
I0213 15:56:52.447602 22261 network_create.go:284] error running [docker network inspect old-k8s-version-745000]: docker network inspect old-k8s-version-745000: exit status 1
stdout:
[]
stderr:
Error response from daemon: network old-k8s-version-745000 not found
I0213 15:56:52.447612 22261 network_create.go:286] output of [docker network inspect old-k8s-version-745000]: -- stdout --
[]
-- /stdout --
** stderr **
Error response from daemon: network old-k8s-version-745000 not found
** /stderr **
I0213 15:56:52.447752 22261 cli_runner.go:164] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0213 15:56:52.502064 22261 network.go:210] skipping subnet 192.168.49.0/24 that is reserved: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0213 15:56:52.502479 22261 network.go:207] using free private subnet 192.168.58.0/24: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc002146f90}
I0213 15:56:52.502496 22261 network_create.go:124] attempt to create docker network old-k8s-version-745000 192.168.58.0/24 with gateway 192.168.58.1 and MTU of 65535 ...
I0213 15:56:52.502559 22261 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.58.0/24 --gateway=192.168.58.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-745000 old-k8s-version-745000
W0213 15:56:52.554342 22261 cli_runner.go:211] docker network create --driver=bridge --subnet=192.168.58.0/24 --gateway=192.168.58.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-745000 old-k8s-version-745000 returned with exit code 1
W0213 15:56:52.554407 22261 network_create.go:149] failed to create docker network old-k8s-version-745000 192.168.58.0/24 with gateway 192.168.58.1 and mtu of 65535: docker network create --driver=bridge --subnet=192.168.58.0/24 --gateway=192.168.58.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-745000 old-k8s-version-745000: exit status 1
stdout:
stderr:
Error response from daemon: Pool overlaps with other one on this address space
W0213 15:56:52.554429 22261 network_create.go:116] failed to create docker network old-k8s-version-745000 192.168.58.0/24, will retry: subnet is taken
I0213 15:56:52.555831 22261 network.go:210] skipping subnet 192.168.58.0/24 that is reserved: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0213 15:56:52.556184 22261 network.go:207] using free private subnet 192.168.67.0/24: &{IP:192.168.67.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.67.0/24 Gateway:192.168.67.1 ClientMin:192.168.67.2 ClientMax:192.168.67.254 Broadcast:192.168.67.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc00239d570}
I0213 15:56:52.556199 22261 network_create.go:124] attempt to create docker network old-k8s-version-745000 192.168.67.0/24 with gateway 192.168.67.1 and MTU of 65535 ...
I0213 15:56:52.556267 22261 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.67.0/24 --gateway=192.168.67.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-745000 old-k8s-version-745000
I0213 15:56:52.644995 22261 network_create.go:108] docker network old-k8s-version-745000 192.168.67.0/24 created
I0213 15:56:52.645043 22261 kic.go:121] calculated static IP "192.168.67.2" for the "old-k8s-version-745000" container
I0213 15:56:52.645152 22261 cli_runner.go:164] Run: docker ps -a --format {{.Names}}
I0213 15:56:52.696629 22261 cli_runner.go:164] Run: docker volume create old-k8s-version-745000 --label name.minikube.sigs.k8s.io=old-k8s-version-745000 --label created_by.minikube.sigs.k8s.io=true
I0213 15:56:52.748320 22261 oci.go:103] Successfully created a docker volume old-k8s-version-745000
I0213 15:56:52.748433 22261 cli_runner.go:164] Run: docker run --rm --name old-k8s-version-745000-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-745000 --entrypoint /usr/bin/test -v old-k8s-version-745000:/var gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 -d /var/lib
I0213 15:56:53.370068 22261 oci.go:107] Successfully prepared a docker volume old-k8s-version-745000
I0213 15:56:53.370105 22261 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0213 15:56:53.370120 22261 kic.go:194] Starting extracting preloaded images to volume ...
I0213 15:56:53.370217 22261 cli_runner.go:164] Run: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/18169-6320/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-745000:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 -I lz4 -xf /preloaded.tar -C /extractDir
I0213 15:56:55.694906 22261 cli_runner.go:217] Completed: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/18169-6320/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-745000:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 -I lz4 -xf /preloaded.tar -C /extractDir: (2.324681886s)
I0213 15:56:55.694936 22261 kic.go:203] duration metric: took 2.324868 seconds to extract preloaded images to volume
I0213 15:56:55.695036 22261 cli_runner.go:164] Run: docker info --format "'{{json .SecurityOptions}}'"
I0213 15:56:55.814238 22261 cli_runner.go:164] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname old-k8s-version-745000 --name old-k8s-version-745000 --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-745000 --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=old-k8s-version-745000 --network old-k8s-version-745000 --ip 192.168.67.2 --volume old-k8s-version-745000:/var --security-opt apparmor=unconfined --memory=2200mb --memory-swap=2200mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0
I0213 15:56:56.245349 22261 cli_runner.go:164] Run: docker container inspect old-k8s-version-745000 --format={{.State.Running}}
I0213 15:56:56.312042 22261 cli_runner.go:164] Run: docker container inspect old-k8s-version-745000 --format={{.State.Status}}
I0213 15:56:56.375876 22261 cli_runner.go:164] Run: docker exec old-k8s-version-745000 stat /var/lib/dpkg/alternatives/iptables
I0213 15:56:56.523853 22261 oci.go:144] the created container "old-k8s-version-745000" has a running status.
I0213 15:56:56.523888 22261 kic.go:225] Creating ssh key for kic: /Users/jenkins/minikube-integration/18169-6320/.minikube/machines/old-k8s-version-745000/id_rsa...
I0213 15:56:56.693061 22261 kic_runner.go:191] docker (temp): /Users/jenkins/minikube-integration/18169-6320/.minikube/machines/old-k8s-version-745000/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0213 15:56:56.773945 22261 cli_runner.go:164] Run: docker container inspect old-k8s-version-745000 --format={{.State.Status}}
I0213 15:56:56.840333 22261 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0213 15:56:56.840356 22261 kic_runner.go:114] Args: [docker exec --privileged old-k8s-version-745000 chown docker:docker /home/docker/.ssh/authorized_keys]
I0213 15:56:56.960329 22261 cli_runner.go:164] Run: docker container inspect old-k8s-version-745000 --format={{.State.Status}}
I0213 15:56:57.017692 22261 machine.go:88] provisioning docker machine ...
I0213 15:56:57.017740 22261 ubuntu.go:169] provisioning hostname "old-k8s-version-745000"
I0213 15:56:57.017840 22261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-745000
I0213 15:56:57.072213 22261 main.go:141] libmachine: Using SSH client type: native
I0213 15:56:57.072545 22261 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1407080] 0x1409d60 <nil> [] 0s} 127.0.0.1 56464 <nil> <nil>}
I0213 15:56:57.072560 22261 main.go:141] libmachine: About to run SSH command:
sudo hostname old-k8s-version-745000 && echo "old-k8s-version-745000" | sudo tee /etc/hostname
I0213 15:56:57.241884 22261 main.go:141] libmachine: SSH cmd err, output: <nil>: old-k8s-version-745000
I0213 15:56:57.241984 22261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-745000
I0213 15:56:57.299540 22261 main.go:141] libmachine: Using SSH client type: native
I0213 15:56:57.299862 22261 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1407080] 0x1409d60 <nil> [] 0s} 127.0.0.1 56464 <nil> <nil>}
I0213 15:56:57.299876 22261 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sold-k8s-version-745000' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 old-k8s-version-745000/g' /etc/hosts;
else
echo '127.0.1.1 old-k8s-version-745000' | sudo tee -a /etc/hosts;
fi
fi
I0213 15:56:57.447707 22261 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0213 15:56:57.447727 22261 ubuntu.go:175] set auth options {CertDir:/Users/jenkins/minikube-integration/18169-6320/.minikube CaCertPath:/Users/jenkins/minikube-integration/18169-6320/.minikube/certs/ca.pem CaPrivateKeyPath:/Users/jenkins/minikube-integration/18169-6320/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/Users/jenkins/minikube-integration/18169-6320/.minikube/machines/server.pem ServerKeyPath:/Users/jenkins/minikube-integration/18169-6320/.minikube/machines/server-key.pem ClientKeyPath:/Users/jenkins/minikube-integration/18169-6320/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/Users/jenkins/minikube-integration/18169-6320/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/Users/jenkins/minikube-integration/18169-6320/.minikube}
I0213 15:56:57.447745 22261 ubuntu.go:177] setting up certificates
I0213 15:56:57.447753 22261 provision.go:83] configureAuth start
I0213 15:56:57.447828 22261 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-745000
I0213 15:56:57.507056 22261 provision.go:138] copyHostCerts
I0213 15:56:57.507191 22261 exec_runner.go:144] found /Users/jenkins/minikube-integration/18169-6320/.minikube/ca.pem, removing ...
I0213 15:56:57.507207 22261 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/18169-6320/.minikube/ca.pem
I0213 15:56:57.507360 22261 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/18169-6320/.minikube/certs/ca.pem --> /Users/jenkins/minikube-integration/18169-6320/.minikube/ca.pem (1078 bytes)
I0213 15:56:57.507606 22261 exec_runner.go:144] found /Users/jenkins/minikube-integration/18169-6320/.minikube/cert.pem, removing ...
I0213 15:56:57.507614 22261 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/18169-6320/.minikube/cert.pem
I0213 15:56:57.507748 22261 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/18169-6320/.minikube/certs/cert.pem --> /Users/jenkins/minikube-integration/18169-6320/.minikube/cert.pem (1123 bytes)
I0213 15:56:57.507943 22261 exec_runner.go:144] found /Users/jenkins/minikube-integration/18169-6320/.minikube/key.pem, removing ...
I0213 15:56:57.507950 22261 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/18169-6320/.minikube/key.pem
I0213 15:56:57.508023 22261 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/18169-6320/.minikube/certs/key.pem --> /Users/jenkins/minikube-integration/18169-6320/.minikube/key.pem (1675 bytes)
I0213 15:56:57.508177 22261 provision.go:112] generating server cert: /Users/jenkins/minikube-integration/18169-6320/.minikube/machines/server.pem ca-key=/Users/jenkins/minikube-integration/18169-6320/.minikube/certs/ca.pem private-key=/Users/jenkins/minikube-integration/18169-6320/.minikube/certs/ca-key.pem org=jenkins.old-k8s-version-745000 san=[192.168.67.2 127.0.0.1 localhost 127.0.0.1 minikube old-k8s-version-745000]
I0213 15:56:57.706994 22261 provision.go:172] copyRemoteCerts
I0213 15:56:57.707055 22261 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0213 15:56:57.707119 22261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-745000
I0213 15:56:57.766067 22261 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:56464 SSHKeyPath:/Users/jenkins/minikube-integration/18169-6320/.minikube/machines/old-k8s-version-745000/id_rsa Username:docker}
I0213 15:56:57.872309 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1078 bytes)
I0213 15:56:57.920029 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/machines/server.pem --> /etc/docker/server.pem (1241 bytes)
I0213 15:56:57.965962 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1675 bytes)
I0213 15:56:58.011191 22261 provision.go:86] duration metric: configureAuth took 563.437331ms
I0213 15:56:58.011223 22261 ubuntu.go:193] setting minikube options for container-runtime
I0213 15:56:58.011355 22261 config.go:182] Loaded profile config "old-k8s-version-745000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.16.0
I0213 15:56:58.011417 22261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-745000
I0213 15:56:58.067204 22261 main.go:141] libmachine: Using SSH client type: native
I0213 15:56:58.067516 22261 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1407080] 0x1409d60 <nil> [] 0s} 127.0.0.1 56464 <nil> <nil>}
I0213 15:56:58.067534 22261 main.go:141] libmachine: About to run SSH command:
df --output=fstype / | tail -n 1
I0213 15:56:58.207166 22261 main.go:141] libmachine: SSH cmd err, output: <nil>: overlay
I0213 15:56:58.207186 22261 ubuntu.go:71] root file system type: overlay
I0213 15:56:58.207314 22261 provision.go:309] Updating docker unit: /lib/systemd/system/docker.service ...
I0213 15:56:58.207411 22261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-745000
I0213 15:56:58.266883 22261 main.go:141] libmachine: Using SSH client type: native
I0213 15:56:58.267231 22261 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1407080] 0x1409d60 <nil> [] 0s} 127.0.0.1 56464 <nil> <nil>}
I0213 15:56:58.267287 22261 main.go:141] libmachine: About to run SSH command:
sudo mkdir -p /lib/systemd/system && printf %s "[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP \$MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service.new
I0213 15:56:58.432375 22261 main.go:141] libmachine: SSH cmd err, output: <nil>: [Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
I0213 15:56:58.432558 22261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-745000
I0213 15:56:58.490858 22261 main.go:141] libmachine: Using SSH client type: native
I0213 15:56:58.491162 22261 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1407080] 0x1409d60 <nil> [] 0s} 127.0.0.1 56464 <nil> <nil>}
I0213 15:56:58.491175 22261 main.go:141] libmachine: About to run SSH command:
sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
I0213 15:56:59.176289 22261 main.go:141] libmachine: SSH cmd err, output: <nil>: --- /lib/systemd/system/docker.service 2023-10-26 09:06:22.000000000 +0000
+++ /lib/systemd/system/docker.service.new 2024-02-13 23:56:58.427230787 +0000
@@ -1,30 +1,32 @@
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
-After=network-online.target docker.socket firewalld.service containerd.service time-set.target
-Wants=network-online.target containerd.service
+BindsTo=containerd.service
+After=network-online.target firewalld.service containerd.service
+Wants=network-online.target
Requires=docker.socket
+StartLimitBurst=3
+StartLimitIntervalSec=60
[Service]
Type=notify
-# the default is not to use systemd for cgroups because the delegate issues still
-# exists and systemd currently does not support the cgroup feature set required
-# for containers run by docker
-ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
-ExecReload=/bin/kill -s HUP $MAINPID
-TimeoutStartSec=0
-RestartSec=2
-Restart=always
+Restart=on-failure
-# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
-# Both the old, and new location are accepted by systemd 229 and up, so using the old location
-# to make them work for either version of systemd.
-StartLimitBurst=3
-# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
-# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
-# this option work for either version of systemd.
-StartLimitInterval=60s
+
+# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
+# The base configuration already specifies an 'ExecStart=...' command. The first directive
+# here is to clear out that command inherited from the base configuration. Without this,
+# the command from the base configuration and the command specified here are treated as
+# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
+# will catch this invalid input and refuse to start the service with an error like:
+# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
+
+# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
+# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
+ExecStart=
+ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
+ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
@@ -32,16 +34,16 @@
LimitNPROC=infinity
LimitCORE=infinity
-# Comment TasksMax if your systemd version does not support it.
-# Only systemd 226 and above support this option.
+# Uncomment TasksMax if your systemd version supports it.
+# Only systemd 226 and above support this version.
TasksMax=infinity
+TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
-OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
I0213 15:56:59.176321 22261 machine.go:91] provisioned docker machine in 2.158651629s
I0213 15:56:59.176331 22261 client.go:171] LocalClient.Create took 6.855200692s
I0213 15:56:59.176352 22261 start.go:167] duration metric: libmachine.API.Create for "old-k8s-version-745000" took 6.85528307s
I0213 15:56:59.176362 22261 start.go:300] post-start starting for "old-k8s-version-745000" (driver="docker")
I0213 15:56:59.176370 22261 start.go:329] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0213 15:56:59.176452 22261 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0213 15:56:59.176544 22261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-745000
I0213 15:56:59.234818 22261 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:56464 SSHKeyPath:/Users/jenkins/minikube-integration/18169-6320/.minikube/machines/old-k8s-version-745000/id_rsa Username:docker}
I0213 15:56:59.341567 22261 ssh_runner.go:195] Run: cat /etc/os-release
I0213 15:56:59.346795 22261 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0213 15:56:59.346834 22261 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0213 15:56:59.346842 22261 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0213 15:56:59.346848 22261 info.go:137] Remote host: Ubuntu 22.04.3 LTS
I0213 15:56:59.346858 22261 filesync.go:126] Scanning /Users/jenkins/minikube-integration/18169-6320/.minikube/addons for local assets ...
I0213 15:56:59.346981 22261 filesync.go:126] Scanning /Users/jenkins/minikube-integration/18169-6320/.minikube/files for local assets ...
I0213 15:56:59.347191 22261 filesync.go:149] local asset: /Users/jenkins/minikube-integration/18169-6320/.minikube/files/etc/ssl/certs/67762.pem -> 67762.pem in /etc/ssl/certs
I0213 15:56:59.347468 22261 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0213 15:56:59.363679 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/files/etc/ssl/certs/67762.pem --> /etc/ssl/certs/67762.pem (1708 bytes)
I0213 15:56:59.404893 22261 start.go:303] post-start completed in 228.506449ms
I0213 15:56:59.405663 22261 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-745000
I0213 15:56:59.482076 22261 profile.go:148] Saving config to /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/config.json ...
I0213 15:56:59.482631 22261 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0213 15:56:59.482701 22261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-745000
I0213 15:56:59.542539 22261 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:56464 SSHKeyPath:/Users/jenkins/minikube-integration/18169-6320/.minikube/machines/old-k8s-version-745000/id_rsa Username:docker}
I0213 15:56:59.636773 22261 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0213 15:56:59.642731 22261 start.go:128] duration metric: createHost completed in 7.343922909s
I0213 15:56:59.642754 22261 start.go:83] releasing machines lock for "old-k8s-version-745000", held for 7.344314573s
I0213 15:56:59.642855 22261 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-745000
I0213 15:56:59.707081 22261 ssh_runner.go:195] Run: cat /version.json
I0213 15:56:59.707080 22261 ssh_runner.go:195] Run: curl -sS -m 2 https://registry.k8s.io/
I0213 15:56:59.707216 22261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-745000
I0213 15:56:59.707275 22261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-745000
I0213 15:56:59.781754 22261 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:56464 SSHKeyPath:/Users/jenkins/minikube-integration/18169-6320/.minikube/machines/old-k8s-version-745000/id_rsa Username:docker}
I0213 15:56:59.781797 22261 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:56464 SSHKeyPath:/Users/jenkins/minikube-integration/18169-6320/.minikube/machines/old-k8s-version-745000/id_rsa Username:docker}
I0213 15:56:59.882012 22261 ssh_runner.go:195] Run: systemctl --version
I0213 15:56:59.993840 22261 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
I0213 15:57:00.001077 22261 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *loopback.conf* -not -name *.mk_disabled -exec sh -c "grep -q loopback {} && ( grep -q name {} || sudo sed -i '/"type": "loopback"/i \ \ \ \ "name": "loopback",' {} ) && sudo sed -i 's|"cniVersion": ".*"|"cniVersion": "1.0.0"|g' {}" ;
I0213 15:57:00.049324 22261 cni.go:230] loopback cni configuration patched: "/etc/cni/net.d/*loopback.conf*" found
I0213 15:57:00.049412 22261 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *bridge* -not -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e '/"dst": ".*:.*"/d' -e 's|^(.*)"dst": (.*)[,*]$|\1"dst": \2|g' -e '/"subnet": ".*:.*"/d' -e 's|^(.*)"subnet": ".*"(.*)[,*]$|\1"subnet": "10.244.0.0/16"\2|g' {}" ;
I0213 15:57:00.084158 22261 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e 's|^(.*)"subnet": ".*"(.*)$|\1"subnet": "10.244.0.0/16"\2|g' -e 's|^(.*)"gateway": ".*"(.*)$|\1"gateway": "10.244.0.1"\2|g' {}" ;
I0213 15:57:00.117394 22261 cni.go:308] configured [/etc/cni/net.d/100-crio-bridge.conf, /etc/cni/net.d/87-podman-bridge.conflist] bridge cni config(s)
I0213 15:57:00.117412 22261 start.go:475] detecting cgroup driver to use...
I0213 15:57:00.117424 22261 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0213 15:57:00.117517 22261 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0213 15:57:00.150453 22261 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.1"|' /etc/containerd/config.toml"
I0213 15:57:00.169437 22261 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I0213 15:57:00.187255 22261 containerd.go:146] configuring containerd to use "cgroupfs" as cgroup driver...
I0213 15:57:00.187340 22261 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = false|g' /etc/containerd/config.toml"
I0213 15:57:00.206407 22261 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0213 15:57:00.224843 22261 ssh_runner.go:195] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I0213 15:57:00.243436 22261 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0213 15:57:00.263176 22261 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0213 15:57:00.280173 22261 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I0213 15:57:00.297403 22261 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0213 15:57:00.313966 22261 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0213 15:57:00.330779 22261 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0213 15:57:00.402695 22261 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0213 15:57:00.493908 22261 start.go:475] detecting cgroup driver to use...
I0213 15:57:00.493928 22261 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0213 15:57:00.493993 22261 ssh_runner.go:195] Run: sudo systemctl cat docker.service
I0213 15:57:00.512890 22261 cruntime.go:279] skipping containerd shutdown because we are bound to it
I0213 15:57:00.512960 22261 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0213 15:57:00.533028 22261 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///var/run/dockershim.sock
" | sudo tee /etc/crictl.yaml"
I0213 15:57:00.571015 22261 ssh_runner.go:195] Run: which cri-dockerd
I0213 15:57:00.575512 22261 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/cri-docker.service.d
I0213 15:57:00.591990 22261 ssh_runner.go:362] scp memory --> /etc/systemd/system/cri-docker.service.d/10-cni.conf (189 bytes)
I0213 15:57:00.627187 22261 ssh_runner.go:195] Run: sudo systemctl unmask docker.service
I0213 15:57:00.700796 22261 ssh_runner.go:195] Run: sudo systemctl enable docker.socket
I0213 15:57:00.808594 22261 docker.go:574] configuring docker to use "cgroupfs" as cgroup driver...
I0213 15:57:00.808681 22261 ssh_runner.go:362] scp memory --> /etc/docker/daemon.json (130 bytes)
I0213 15:57:00.846395 22261 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0213 15:57:00.920920 22261 ssh_runner.go:195] Run: sudo systemctl restart docker
I0213 15:57:01.288726 22261 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0213 15:57:01.315299 22261 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0213 15:57:01.384809 22261 out.go:204] * Preparing Kubernetes v1.16.0 on Docker 24.0.7 ...
I0213 15:57:01.384946 22261 cli_runner.go:164] Run: docker exec -t old-k8s-version-745000 dig +short host.docker.internal
I0213 15:57:01.525465 22261 network.go:96] got host ip for mount in container by digging dns: 192.168.65.254
I0213 15:57:01.525564 22261 ssh_runner.go:195] Run: grep 192.168.65.254 host.minikube.internal$ /etc/hosts
I0213 15:57:01.530334 22261 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.65.254 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0213 15:57:01.547791 22261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "8443/tcp") 0).HostPort}}'" old-k8s-version-745000
I0213 15:57:01.605569 22261 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0213 15:57:01.605652 22261 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0213 15:57:01.627262 22261 docker.go:685] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0213 15:57:01.627295 22261 docker.go:691] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0213 15:57:01.627383 22261 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0213 15:57:01.645386 22261 ssh_runner.go:195] Run: which lz4
I0213 15:57:01.650371 22261 ssh_runner.go:195] Run: stat -c "%s %y" /preloaded.tar.lz4
I0213 15:57:01.654739 22261 ssh_runner.go:352] existence check for /preloaded.tar.lz4: stat -c "%s %y" /preloaded.tar.lz4: Process exited with status 1
stdout:
stderr:
stat: cannot statx '/preloaded.tar.lz4': No such file or directory
I0213 15:57:01.654764 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 --> /preloaded.tar.lz4 (369789069 bytes)
I0213 15:57:08.556470 22261 docker.go:649] Took 6.906283 seconds to copy over tarball
I0213 15:57:08.556546 22261 ssh_runner.go:195] Run: sudo tar --xattrs --xattrs-include security.capability -I lz4 -C /var -xf /preloaded.tar.lz4
I0213 15:57:10.361204 22261 ssh_runner.go:235] Completed: sudo tar --xattrs --xattrs-include security.capability -I lz4 -C /var -xf /preloaded.tar.lz4: (1.804658418s)
I0213 15:57:10.361234 22261 ssh_runner.go:146] rm: /preloaded.tar.lz4
I0213 15:57:10.410977 22261 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0213 15:57:10.427813 22261 ssh_runner.go:362] scp memory --> /var/lib/docker/image/overlay2/repositories.json (2499 bytes)
I0213 15:57:10.457791 22261 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0213 15:57:10.526862 22261 ssh_runner.go:195] Run: sudo systemctl restart docker
I0213 15:57:11.702844 22261 ssh_runner.go:235] Completed: sudo systemctl restart docker: (1.175972459s)
I0213 15:57:11.702984 22261 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0213 15:57:11.728437 22261 docker.go:685] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0213 15:57:11.728450 22261 docker.go:691] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0213 15:57:11.728460 22261 cache_images.go:88] LoadImages start: [registry.k8s.io/kube-apiserver:v1.16.0 registry.k8s.io/kube-controller-manager:v1.16.0 registry.k8s.io/kube-scheduler:v1.16.0 registry.k8s.io/kube-proxy:v1.16.0 registry.k8s.io/pause:3.1 registry.k8s.io/etcd:3.3.15-0 registry.k8s.io/coredns:1.6.2 gcr.io/k8s-minikube/storage-provisioner:v5]
I0213 15:57:11.734181 22261 image.go:134] retrieving image: registry.k8s.io/kube-scheduler:v1.16.0
I0213 15:57:11.734487 22261 image.go:134] retrieving image: registry.k8s.io/kube-controller-manager:v1.16.0
I0213 15:57:11.734827 22261 image.go:134] retrieving image: gcr.io/k8s-minikube/storage-provisioner:v5
I0213 15:57:11.735489 22261 image.go:134] retrieving image: registry.k8s.io/kube-apiserver:v1.16.0
I0213 15:57:11.736032 22261 image.go:134] retrieving image: registry.k8s.io/pause:3.1
I0213 15:57:11.736053 22261 image.go:134] retrieving image: registry.k8s.io/etcd:3.3.15-0
I0213 15:57:11.736127 22261 image.go:134] retrieving image: registry.k8s.io/kube-proxy:v1.16.0
I0213 15:57:11.736266 22261 image.go:134] retrieving image: registry.k8s.io/coredns:1.6.2
I0213 15:57:11.765383 22261 image.go:177] daemon lookup for gcr.io/k8s-minikube/storage-provisioner:v5: Error response from daemon: No such image: gcr.io/k8s-minikube/storage-provisioner:v5
I0213 15:57:11.765442 22261 image.go:177] daemon lookup for registry.k8s.io/kube-controller-manager:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-controller-manager:v1.16.0
I0213 15:57:11.765481 22261 image.go:177] daemon lookup for registry.k8s.io/kube-scheduler:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-scheduler:v1.16.0
I0213 15:57:11.765644 22261 image.go:177] daemon lookup for registry.k8s.io/kube-proxy:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-proxy:v1.16.0
I0213 15:57:11.785360 22261 image.go:177] daemon lookup for registry.k8s.io/coredns:1.6.2: Error response from daemon: No such image: registry.k8s.io/coredns:1.6.2
I0213 15:57:11.785386 22261 image.go:177] daemon lookup for registry.k8s.io/etcd:3.3.15-0: Error response from daemon: No such image: registry.k8s.io/etcd:3.3.15-0
I0213 15:57:11.785471 22261 image.go:177] daemon lookup for registry.k8s.io/pause:3.1: Error response from daemon: No such image: registry.k8s.io/pause:3.1
I0213 15:57:11.785467 22261 image.go:177] daemon lookup for registry.k8s.io/kube-apiserver:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-apiserver:v1.16.0
I0213 15:57:13.752680 22261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-scheduler:v1.16.0
I0213 15:57:13.773412 22261 cache_images.go:116] "registry.k8s.io/kube-scheduler:v1.16.0" needs transfer: "registry.k8s.io/kube-scheduler:v1.16.0" does not exist at hash "301ddc62b80b16315d3c2653cf3888370394277afb3187614cfa20edc352ca0a" in container runtime
I0213 15:57:13.773448 22261 docker.go:337] Removing image: registry.k8s.io/kube-scheduler:v1.16.0
I0213 15:57:13.773504 22261 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-scheduler:v1.16.0
I0213 15:57:13.792770 22261 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/18169-6320/.minikube/cache/images/amd64/registry.k8s.io/kube-scheduler_v1.16.0
I0213 15:57:13.830207 22261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/coredns:1.6.2
I0213 15:57:13.849634 22261 cache_images.go:116] "registry.k8s.io/coredns:1.6.2" needs transfer: "registry.k8s.io/coredns:1.6.2" does not exist at hash "bf261d157914477ee1a5969d28ec687f3fbfc9fb5a664b22df78e57023b0e03b" in container runtime
I0213 15:57:13.849660 22261 docker.go:337] Removing image: registry.k8s.io/coredns:1.6.2
I0213 15:57:13.849738 22261 ssh_runner.go:195] Run: docker rmi registry.k8s.io/coredns:1.6.2
I0213 15:57:13.861605 22261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/etcd:3.3.15-0
I0213 15:57:13.869464 22261 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/18169-6320/.minikube/cache/images/amd64/registry.k8s.io/coredns_1.6.2
I0213 15:57:13.879357 22261 cache_images.go:116] "registry.k8s.io/etcd:3.3.15-0" needs transfer: "registry.k8s.io/etcd:3.3.15-0" does not exist at hash "b2756210eeabf84f3221da9959e9483f3919dc2aaab4cd45e7cd072fcbde27ed" in container runtime
I0213 15:57:13.879384 22261 docker.go:337] Removing image: registry.k8s.io/etcd:3.3.15-0
I0213 15:57:13.879444 22261 ssh_runner.go:195] Run: docker rmi registry.k8s.io/etcd:3.3.15-0
I0213 15:57:13.894515 22261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-apiserver:v1.16.0
I0213 15:57:13.894946 22261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/pause:3.1
I0213 15:57:13.899212 22261 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/18169-6320/.minikube/cache/images/amd64/registry.k8s.io/etcd_3.3.15-0
I0213 15:57:13.899222 22261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-proxy:v1.16.0
I0213 15:57:13.904581 22261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-controller-manager:v1.16.0
I0213 15:57:13.921454 22261 cache_images.go:116] "registry.k8s.io/pause:3.1" needs transfer: "registry.k8s.io/pause:3.1" does not exist at hash "da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e" in container runtime
I0213 15:57:13.921490 22261 docker.go:337] Removing image: registry.k8s.io/pause:3.1
I0213 15:57:13.921492 22261 cache_images.go:116] "registry.k8s.io/kube-apiserver:v1.16.0" needs transfer: "registry.k8s.io/kube-apiserver:v1.16.0" does not exist at hash "b305571ca60a5a7818bda47da122683d75e8a1907475681ee8b1efbd06bff12e" in container runtime
I0213 15:57:13.921518 22261 docker.go:337] Removing image: registry.k8s.io/kube-apiserver:v1.16.0
I0213 15:57:13.921571 22261 ssh_runner.go:195] Run: docker rmi registry.k8s.io/pause:3.1
I0213 15:57:13.921613 22261 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-apiserver:v1.16.0
I0213 15:57:13.925576 22261 cache_images.go:116] "registry.k8s.io/kube-proxy:v1.16.0" needs transfer: "registry.k8s.io/kube-proxy:v1.16.0" does not exist at hash "c21b0c7400f988db4777858edd13b6d3930d62d7ccf026d2415485a52037f384" in container runtime
I0213 15:57:13.925621 22261 docker.go:337] Removing image: registry.k8s.io/kube-proxy:v1.16.0
I0213 15:57:13.925751 22261 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-proxy:v1.16.0
I0213 15:57:13.928963 22261 cache_images.go:116] "registry.k8s.io/kube-controller-manager:v1.16.0" needs transfer: "registry.k8s.io/kube-controller-manager:v1.16.0" does not exist at hash "06a629a7e51cdcc81a5ed6a3e6650348312f20c954ac52ee489a023628ec9c7d" in container runtime
I0213 15:57:13.929002 22261 docker.go:337] Removing image: registry.k8s.io/kube-controller-manager:v1.16.0
I0213 15:57:13.929087 22261 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-controller-manager:v1.16.0
I0213 15:57:13.987876 22261 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/18169-6320/.minikube/cache/images/amd64/registry.k8s.io/pause_3.1
I0213 15:57:13.987884 22261 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/18169-6320/.minikube/cache/images/amd64/registry.k8s.io/kube-apiserver_v1.16.0
I0213 15:57:13.992576 22261 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/18169-6320/.minikube/cache/images/amd64/registry.k8s.io/kube-proxy_v1.16.0
I0213 15:57:13.998856 22261 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/18169-6320/.minikube/cache/images/amd64/registry.k8s.io/kube-controller-manager_v1.16.0
I0213 15:57:14.079557 22261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} gcr.io/k8s-minikube/storage-provisioner:v5
I0213 15:57:14.100017 22261 cache_images.go:92] LoadImages completed in 2.371596096s
W0213 15:57:14.100073 22261 out.go:239] X Unable to load cached images: loading cached images: stat /Users/jenkins/minikube-integration/18169-6320/.minikube/cache/images/amd64/registry.k8s.io/kube-scheduler_v1.16.0: no such file or directory
X Unable to load cached images: loading cached images: stat /Users/jenkins/minikube-integration/18169-6320/.minikube/cache/images/amd64/registry.k8s.io/kube-scheduler_v1.16.0: no such file or directory
I0213 15:57:14.100153 22261 ssh_runner.go:195] Run: docker info --format {{.CgroupDriver}}
I0213 15:57:14.148405 22261 cni.go:84] Creating CNI manager for ""
I0213 15:57:14.148423 22261 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0213 15:57:14.148436 22261 kubeadm.go:87] Using pod CIDR: 10.244.0.0/16
I0213 15:57:14.148473 22261 kubeadm.go:176] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.67.2 APIServerPort:8443 KubernetesVersion:v1.16.0 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:old-k8s-version-745000 NodeName:old-k8s-version-745000 DNSDomain:cluster.local CRISocket:/var/run/dockershim.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.67.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.67.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticP
odPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[hairpinMode:hairpin-veth runtimeRequestTimeout:15m] PrependCriSocketUnix:false}
I0213 15:57:14.148566 22261 kubeadm.go:181] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.67.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: "old-k8s-version-745000"
kubeletExtraArgs:
node-ip: 192.168.67.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.67.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: old-k8s-version-745000
controlPlaneEndpoint: control-plane.minikube.internal:8443
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
listen-metrics-urls: http://127.0.0.1:2381,http://192.168.67.2:2381
kubernetesVersion: v1.16.0
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0213 15:57:14.148620 22261 kubeadm.go:976] kubelet [Unit]
Wants=docker.socket
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.16.0/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=old-k8s-version-745000 --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.67.2
[Install]
config:
{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-745000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:}
I0213 15:57:14.148682 22261 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.16.0
I0213 15:57:14.163661 22261 binaries.go:44] Found k8s binaries, skipping transfer
I0213 15:57:14.163729 22261 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0213 15:57:14.178700 22261 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (348 bytes)
I0213 15:57:14.208911 22261 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0213 15:57:14.240768 22261 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2174 bytes)
I0213 15:57:14.272688 22261 ssh_runner.go:195] Run: grep 192.168.67.2 control-plane.minikube.internal$ /etc/hosts
I0213 15:57:14.277601 22261 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.67.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0213 15:57:14.295950 22261 certs.go:56] Setting up /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000 for IP: 192.168.67.2
I0213 15:57:14.295969 22261 certs.go:190] acquiring lock for shared ca certs: {Name:mkc037f48c69539d66bd92ede4890b05c28518b3 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 15:57:14.296148 22261 certs.go:199] skipping minikubeCA CA generation: /Users/jenkins/minikube-integration/18169-6320/.minikube/ca.key
I0213 15:57:14.296220 22261 certs.go:199] skipping proxyClientCA CA generation: /Users/jenkins/minikube-integration/18169-6320/.minikube/proxy-client-ca.key
I0213 15:57:14.296263 22261 certs.go:319] generating minikube-user signed cert: /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/client.key
I0213 15:57:14.296274 22261 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/client.crt with IP's: []
I0213 15:57:14.448899 22261 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/client.crt ...
I0213 15:57:14.448911 22261 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/client.crt: {Name:mk68fdcfa92864c3ea08060bdee743ea3e245109 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 15:57:14.459727 22261 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/client.key ...
I0213 15:57:14.459750 22261 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/client.key: {Name:mkf78ccc73ead0e4fef122c0c2bcb200766f4699 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 15:57:14.482178 22261 certs.go:319] generating minikube signed cert: /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/apiserver.key.c7fa3a9e
I0213 15:57:14.482221 22261 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/apiserver.crt.c7fa3a9e with IP's: [192.168.67.2 10.96.0.1 127.0.0.1 10.0.0.1]
I0213 15:57:14.530577 22261 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/apiserver.crt.c7fa3a9e ...
I0213 15:57:14.530588 22261 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/apiserver.crt.c7fa3a9e: {Name:mkd6a0313b6d700ea2153799ade7a3cf02b207f0 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 15:57:14.566670 22261 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/apiserver.key.c7fa3a9e ...
I0213 15:57:14.566735 22261 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/apiserver.key.c7fa3a9e: {Name:mkbeeb2c27d0efc6401a3a7b66a03a47ab506bae Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 15:57:14.587766 22261 certs.go:337] copying /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/apiserver.crt.c7fa3a9e -> /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/apiserver.crt
I0213 15:57:14.588151 22261 certs.go:341] copying /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/apiserver.key.c7fa3a9e -> /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/apiserver.key
I0213 15:57:14.588447 22261 certs.go:319] generating aggregator signed cert: /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/proxy-client.key
I0213 15:57:14.588470 22261 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/proxy-client.crt with IP's: []
I0213 15:57:14.678187 22261 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/proxy-client.crt ...
I0213 15:57:14.678203 22261 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/proxy-client.crt: {Name:mk16e0d45474ebd3450e84601182e06e41fd5acc Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 15:57:14.678553 22261 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/proxy-client.key ...
I0213 15:57:14.678562 22261 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/proxy-client.key: {Name:mk6a3cb3bd4e33cd7fec6b15c08b7de342a942f6 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 15:57:14.679058 22261 certs.go:437] found cert: /Users/jenkins/minikube-integration/18169-6320/.minikube/certs/Users/jenkins/minikube-integration/18169-6320/.minikube/certs/6776.pem (1338 bytes)
W0213 15:57:14.679110 22261 certs.go:433] ignoring /Users/jenkins/minikube-integration/18169-6320/.minikube/certs/Users/jenkins/minikube-integration/18169-6320/.minikube/certs/6776_empty.pem, impossibly tiny 0 bytes
I0213 15:57:14.679120 22261 certs.go:437] found cert: /Users/jenkins/minikube-integration/18169-6320/.minikube/certs/Users/jenkins/minikube-integration/18169-6320/.minikube/certs/ca-key.pem (1675 bytes)
I0213 15:57:14.679205 22261 certs.go:437] found cert: /Users/jenkins/minikube-integration/18169-6320/.minikube/certs/Users/jenkins/minikube-integration/18169-6320/.minikube/certs/ca.pem (1078 bytes)
I0213 15:57:14.679314 22261 certs.go:437] found cert: /Users/jenkins/minikube-integration/18169-6320/.minikube/certs/Users/jenkins/minikube-integration/18169-6320/.minikube/certs/cert.pem (1123 bytes)
I0213 15:57:14.679371 22261 certs.go:437] found cert: /Users/jenkins/minikube-integration/18169-6320/.minikube/certs/Users/jenkins/minikube-integration/18169-6320/.minikube/certs/key.pem (1675 bytes)
I0213 15:57:14.679459 22261 certs.go:437] found cert: /Users/jenkins/minikube-integration/18169-6320/.minikube/files/etc/ssl/certs/Users/jenkins/minikube-integration/18169-6320/.minikube/files/etc/ssl/certs/67762.pem (1708 bytes)
I0213 15:57:14.680023 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1399 bytes)
I0213 15:57:14.739626 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1679 bytes)
I0213 15:57:14.785017 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0213 15:57:14.825438 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/profiles/old-k8s-version-745000/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1679 bytes)
I0213 15:57:14.871148 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0213 15:57:14.930388 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1679 bytes)
I0213 15:57:14.972706 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0213 15:57:15.013443 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1679 bytes)
I0213 15:57:15.055463 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/certs/6776.pem --> /usr/share/ca-certificates/6776.pem (1338 bytes)
I0213 15:57:15.096568 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/files/etc/ssl/certs/67762.pem --> /usr/share/ca-certificates/67762.pem (1708 bytes)
I0213 15:57:15.148543 22261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18169-6320/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0213 15:57:15.195386 22261 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0213 15:57:15.230314 22261 ssh_runner.go:195] Run: openssl version
I0213 15:57:15.236653 22261 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0213 15:57:15.256363 22261 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0213 15:57:15.261457 22261 certs.go:480] hashing: -rw-r--r-- 1 root root 1111 Feb 13 22:54 /usr/share/ca-certificates/minikubeCA.pem
I0213 15:57:15.261509 22261 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0213 15:57:15.269789 22261 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0213 15:57:15.287963 22261 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/6776.pem && ln -fs /usr/share/ca-certificates/6776.pem /etc/ssl/certs/6776.pem"
I0213 15:57:15.306388 22261 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/6776.pem
I0213 15:57:15.312624 22261 certs.go:480] hashing: -rw-r--r-- 1 root root 1338 Feb 13 23:02 /usr/share/ca-certificates/6776.pem
I0213 15:57:15.312676 22261 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/6776.pem
I0213 15:57:15.320384 22261 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/6776.pem /etc/ssl/certs/51391683.0"
I0213 15:57:15.339625 22261 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/67762.pem && ln -fs /usr/share/ca-certificates/67762.pem /etc/ssl/certs/67762.pem"
I0213 15:57:15.358205 22261 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/67762.pem
I0213 15:57:15.364006 22261 certs.go:480] hashing: -rw-r--r-- 1 root root 1708 Feb 13 23:02 /usr/share/ca-certificates/67762.pem
I0213 15:57:15.364081 22261 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/67762.pem
I0213 15:57:15.372703 22261 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/67762.pem /etc/ssl/certs/3ec20f2e.0"
I0213 15:57:15.393503 22261 ssh_runner.go:195] Run: ls /var/lib/minikube/certs/etcd
I0213 15:57:15.400240 22261 certs.go:353] certs directory doesn't exist, likely first start: ls /var/lib/minikube/certs/etcd: Process exited with status 2
stdout:
stderr:
ls: cannot access '/var/lib/minikube/certs/etcd': No such file or directory
I0213 15:57:15.400296 22261 kubeadm.go:404] StartCluster: {Name:old-k8s-version-745000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-745000 Namespace:default APIServerName:minikubeCA APISer
verNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.67.2 Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false Disab
leMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs:}
I0213 15:57:15.400403 22261 ssh_runner.go:195] Run: docker ps --filter status=paused --filter=name=k8s_.*_(kube-system)_ --format={{.ID}}
I0213 15:57:15.422683 22261 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0213 15:57:15.440836 22261 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0213 15:57:15.458664 22261 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0213 15:57:15.458722 22261 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0213 15:57:15.477686 22261 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0213 15:57:15.477719 22261 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0213 15:57:15.584760 22261 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0213 15:57:15.584800 22261 kubeadm.go:322] [preflight] Running pre-flight checks
I0213 15:57:15.868959 22261 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0213 15:57:15.869055 22261 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0213 15:57:15.869137 22261 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0213 15:57:16.056948 22261 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0213 15:57:16.057930 22261 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0213 15:57:16.065146 22261 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0213 15:57:16.152190 22261 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0213 15:57:16.239982 22261 out.go:204] - Generating certificates and keys ...
I0213 15:57:16.240077 22261 kubeadm.go:322] [certs] Using existing ca certificate authority
I0213 15:57:16.240164 22261 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0213 15:57:16.372426 22261 kubeadm.go:322] [certs] Generating "apiserver-kubelet-client" certificate and key
I0213 15:57:16.487364 22261 kubeadm.go:322] [certs] Generating "front-proxy-ca" certificate and key
I0213 15:57:16.652842 22261 kubeadm.go:322] [certs] Generating "front-proxy-client" certificate and key
I0213 15:57:16.824376 22261 kubeadm.go:322] [certs] Generating "etcd/ca" certificate and key
I0213 15:57:16.989096 22261 kubeadm.go:322] [certs] Generating "etcd/server" certificate and key
I0213 15:57:16.989379 22261 kubeadm.go:322] [certs] etcd/server serving cert is signed for DNS names [old-k8s-version-745000 localhost] and IPs [192.168.67.2 127.0.0.1 ::1]
I0213 15:57:17.174766 22261 kubeadm.go:322] [certs] Generating "etcd/peer" certificate and key
I0213 15:57:17.174931 22261 kubeadm.go:322] [certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-745000 localhost] and IPs [192.168.67.2 127.0.0.1 ::1]
I0213 15:57:17.387314 22261 kubeadm.go:322] [certs] Generating "etcd/healthcheck-client" certificate and key
I0213 15:57:17.582444 22261 kubeadm.go:322] [certs] Generating "apiserver-etcd-client" certificate and key
I0213 15:57:17.627734 22261 kubeadm.go:322] [certs] Generating "sa" key and public key
I0213 15:57:17.627976 22261 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0213 15:57:17.698045 22261 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0213 15:57:18.029144 22261 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0213 15:57:18.206083 22261 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0213 15:57:18.564199 22261 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0213 15:57:18.564880 22261 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0213 15:57:18.588831 22261 out.go:204] - Booting up control plane ...
I0213 15:57:18.588913 22261 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0213 15:57:18.589036 22261 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0213 15:57:18.589110 22261 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0213 15:57:18.589182 22261 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0213 15:57:18.589303 22261 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0213 15:57:58.575294 22261 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0213 15:57:58.576158 22261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 15:57:58.576360 22261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 15:58:03.577070 22261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 15:58:03.577234 22261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 15:58:13.578972 22261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 15:58:13.579139 22261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 15:58:33.579861 22261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 15:58:33.580077 22261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 15:59:13.580658 22261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 15:59:13.580845 22261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 15:59:13.580856 22261 kubeadm.go:322]
I0213 15:59:13.580887 22261 kubeadm.go:322] Unfortunately, an error has occurred:
I0213 15:59:13.580926 22261 kubeadm.go:322] timed out waiting for the condition
I0213 15:59:13.580939 22261 kubeadm.go:322]
I0213 15:59:13.580974 22261 kubeadm.go:322] This error is likely caused by:
I0213 15:59:13.581007 22261 kubeadm.go:322] - The kubelet is not running
I0213 15:59:13.581089 22261 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0213 15:59:13.581096 22261 kubeadm.go:322]
I0213 15:59:13.581176 22261 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0213 15:59:13.581213 22261 kubeadm.go:322] - 'systemctl status kubelet'
I0213 15:59:13.581244 22261 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0213 15:59:13.581251 22261 kubeadm.go:322]
I0213 15:59:13.581366 22261 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0213 15:59:13.581498 22261 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0213 15:59:13.581629 22261 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0213 15:59:13.581671 22261 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0213 15:59:13.581732 22261 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0213 15:59:13.581762 22261 kubeadm.go:322] - 'docker logs CONTAINERID'
I0213 15:59:13.586215 22261 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0213 15:59:13.586287 22261 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0213 15:59:13.586456 22261 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
I0213 15:59:13.586599 22261 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0213 15:59:13.586718 22261 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0213 15:59:13.586818 22261 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
W0213 15:59:13.586901 22261 out.go:239] ! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-745000 localhost] and IPs [192.168.67.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-745000 localhost] and IPs [192.168.67.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-745000 localhost] and IPs [192.168.67.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-745000 localhost] and IPs [192.168.67.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
I0213 15:59:13.586947 22261 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm reset --cri-socket /var/run/dockershim.sock --force"
I0213 15:59:14.068026 22261 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service kubelet
I0213 15:59:14.085129 22261 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0213 15:59:14.085185 22261 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0213 15:59:14.100223 22261 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0213 15:59:14.100249 22261 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0213 15:59:14.154700 22261 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0213 15:59:14.154767 22261 kubeadm.go:322] [preflight] Running pre-flight checks
I0213 15:59:14.429273 22261 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0213 15:59:14.429370 22261 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0213 15:59:14.429451 22261 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0213 15:59:14.640801 22261 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0213 15:59:14.641728 22261 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0213 15:59:14.648561 22261 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0213 15:59:14.710170 22261 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0213 15:59:14.752480 22261 out.go:204] - Generating certificates and keys ...
I0213 15:59:14.752569 22261 kubeadm.go:322] [certs] Using existing ca certificate authority
I0213 15:59:14.752640 22261 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0213 15:59:14.752708 22261 kubeadm.go:322] [certs] Using existing apiserver-kubelet-client certificate and key on disk
I0213 15:59:14.752769 22261 kubeadm.go:322] [certs] Using existing front-proxy-ca certificate authority
I0213 15:59:14.752835 22261 kubeadm.go:322] [certs] Using existing front-proxy-client certificate and key on disk
I0213 15:59:14.752902 22261 kubeadm.go:322] [certs] Using existing etcd/ca certificate authority
I0213 15:59:14.752962 22261 kubeadm.go:322] [certs] Using existing etcd/server certificate and key on disk
I0213 15:59:14.753019 22261 kubeadm.go:322] [certs] Using existing etcd/peer certificate and key on disk
I0213 15:59:14.753093 22261 kubeadm.go:322] [certs] Using existing etcd/healthcheck-client certificate and key on disk
I0213 15:59:14.753152 22261 kubeadm.go:322] [certs] Using existing apiserver-etcd-client certificate and key on disk
I0213 15:59:14.753179 22261 kubeadm.go:322] [certs] Using the existing "sa" key
I0213 15:59:14.753221 22261 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0213 15:59:14.770772 22261 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0213 15:59:14.858088 22261 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0213 15:59:14.929882 22261 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0213 15:59:15.071136 22261 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0213 15:59:15.072015 22261 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0213 15:59:15.093540 22261 out.go:204] - Booting up control plane ...
I0213 15:59:15.093651 22261 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0213 15:59:15.093726 22261 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0213 15:59:15.093788 22261 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0213 15:59:15.093849 22261 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0213 15:59:15.093982 22261 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0213 15:59:55.083845 22261 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0213 15:59:55.084752 22261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 15:59:55.084936 22261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 16:00:00.086724 22261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 16:00:00.086922 22261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 16:00:10.087458 22261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 16:00:10.087640 22261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 16:00:30.088592 22261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 16:00:30.088788 22261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 16:01:10.089568 22261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 16:01:10.089783 22261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 16:01:10.089802 22261 kubeadm.go:322]
I0213 16:01:10.089875 22261 kubeadm.go:322] Unfortunately, an error has occurred:
I0213 16:01:10.089922 22261 kubeadm.go:322] timed out waiting for the condition
I0213 16:01:10.089929 22261 kubeadm.go:322]
I0213 16:01:10.089957 22261 kubeadm.go:322] This error is likely caused by:
I0213 16:01:10.089990 22261 kubeadm.go:322] - The kubelet is not running
I0213 16:01:10.090071 22261 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0213 16:01:10.090078 22261 kubeadm.go:322]
I0213 16:01:10.090164 22261 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0213 16:01:10.090189 22261 kubeadm.go:322] - 'systemctl status kubelet'
I0213 16:01:10.090220 22261 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0213 16:01:10.090227 22261 kubeadm.go:322]
I0213 16:01:10.090313 22261 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0213 16:01:10.090389 22261 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0213 16:01:10.090460 22261 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0213 16:01:10.090495 22261 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0213 16:01:10.090560 22261 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0213 16:01:10.090601 22261 kubeadm.go:322] - 'docker logs CONTAINERID'
I0213 16:01:10.095257 22261 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0213 16:01:10.095331 22261 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0213 16:01:10.095437 22261 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
I0213 16:01:10.095526 22261 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0213 16:01:10.095592 22261 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0213 16:01:10.095648 22261 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
I0213 16:01:10.095677 22261 kubeadm.go:406] StartCluster complete in 3m54.700458975s
I0213 16:01:10.095757 22261 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-apiserver --format={{.ID}}
I0213 16:01:10.114625 22261 logs.go:276] 0 containers: []
W0213 16:01:10.114639 22261 logs.go:278] No container was found matching "kube-apiserver"
I0213 16:01:10.114702 22261 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_etcd --format={{.ID}}
I0213 16:01:10.135046 22261 logs.go:276] 0 containers: []
W0213 16:01:10.135060 22261 logs.go:278] No container was found matching "etcd"
I0213 16:01:10.135126 22261 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_coredns --format={{.ID}}
I0213 16:01:10.152261 22261 logs.go:276] 0 containers: []
W0213 16:01:10.152275 22261 logs.go:278] No container was found matching "coredns"
I0213 16:01:10.152346 22261 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-scheduler --format={{.ID}}
I0213 16:01:10.170655 22261 logs.go:276] 0 containers: []
W0213 16:01:10.170669 22261 logs.go:278] No container was found matching "kube-scheduler"
I0213 16:01:10.170736 22261 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-proxy --format={{.ID}}
I0213 16:01:10.189888 22261 logs.go:276] 0 containers: []
W0213 16:01:10.189902 22261 logs.go:278] No container was found matching "kube-proxy"
I0213 16:01:10.189965 22261 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-controller-manager --format={{.ID}}
I0213 16:01:10.208544 22261 logs.go:276] 0 containers: []
W0213 16:01:10.208558 22261 logs.go:278] No container was found matching "kube-controller-manager"
I0213 16:01:10.208629 22261 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kindnet --format={{.ID}}
I0213 16:01:10.227784 22261 logs.go:276] 0 containers: []
W0213 16:01:10.227801 22261 logs.go:278] No container was found matching "kindnet"
I0213 16:01:10.227810 22261 logs.go:123] Gathering logs for describe nodes ...
I0213 16:01:10.227820 22261 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0213 16:01:10.298369 22261 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0213 16:01:10.298381 22261 logs.go:123] Gathering logs for Docker ...
I0213 16:01:10.298388 22261 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u docker -u cri-docker -n 400"
I0213 16:01:10.320291 22261 logs.go:123] Gathering logs for container status ...
I0213 16:01:10.320306 22261 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0213 16:01:10.381755 22261 logs.go:123] Gathering logs for kubelet ...
I0213 16:01:10.381769 22261 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
I0213 16:01:10.426982 22261 logs.go:123] Gathering logs for dmesg ...
I0213 16:01:10.426999 22261 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
W0213 16:01:10.448419 22261 out.go:364] Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0213 16:01:10.448439 22261 out.go:239] *
*
W0213 16:01:10.448489 22261 out.go:239] X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0213 16:01:10.448512 22261 out.go:239] *
*
W0213 16:01:10.449252 22261 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0213 16:01:10.512909 22261 out.go:177]
W0213 16:01:10.533668 22261 out.go:239] X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0213 16:01:10.533701 22261 out.go:239] * Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
* Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
W0213 16:01:10.533725 22261 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/4172
* Related issue: https://github.com/kubernetes/minikube/issues/4172
I0213 16:01:10.554758 22261 out.go:177]
** /stderr **
start_stop_delete_test.go:188: failed starting minikube -first start-. args "out/minikube-darwin-amd64 start -p old-k8s-version-745000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0": exit status 109
helpers_test.go:222: -----------------------post-mortem--------------------------------
helpers_test.go:230: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/FirstStart]: docker inspect <======
helpers_test.go:231: (dbg) Run: docker inspect old-k8s-version-745000
helpers_test.go:235: (dbg) docker inspect old-k8s-version-745000:
-- stdout --
[
{
"Id": "2b4f372aa2467d6d5f725ec22664a522f6dc91951f9b73b544495bf22b4e63b7",
"Created": "2024-02-13T23:56:55.870618044Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 356690,
"ExitCode": 0,
"Error": "",
"StartedAt": "2024-02-13T23:56:56.188240128Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:9941de2e064a4a6a7155bfc66cedd2854b8c725b77bb8d4eaf81bef39f951dd7",
"ResolvConfPath": "/var/lib/docker/containers/2b4f372aa2467d6d5f725ec22664a522f6dc91951f9b73b544495bf22b4e63b7/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/2b4f372aa2467d6d5f725ec22664a522f6dc91951f9b73b544495bf22b4e63b7/hostname",
"HostsPath": "/var/lib/docker/containers/2b4f372aa2467d6d5f725ec22664a522f6dc91951f9b73b544495bf22b4e63b7/hosts",
"LogPath": "/var/lib/docker/containers/2b4f372aa2467d6d5f725ec22664a522f6dc91951f9b73b544495bf22b4e63b7/2b4f372aa2467d6d5f725ec22664a522f6dc91951f9b73b544495bf22b4e63b7-json.log",
"Name": "/old-k8s-version-745000",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro",
"old-k8s-version-745000:/var"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "old-k8s-version-745000",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 2306867200,
"NanoCpus": 2000000000,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": [],
"BlkioDeviceWriteBps": [],
"BlkioDeviceReadIOps": [],
"BlkioDeviceWriteIOps": [],
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 2306867200,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": [],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/3b6060eec3a1586a55a82fcdff198c231a72ae85ee6f6b4c6b13c556de95556d-init/diff:/var/lib/docker/overlay2/17d01b22a52da825ae58e67decfe3f4c8ae2f6fe80510c1be556e233e058ce7a/diff",
"MergedDir": "/var/lib/docker/overlay2/3b6060eec3a1586a55a82fcdff198c231a72ae85ee6f6b4c6b13c556de95556d/merged",
"UpperDir": "/var/lib/docker/overlay2/3b6060eec3a1586a55a82fcdff198c231a72ae85ee6f6b4c6b13c556de95556d/diff",
"WorkDir": "/var/lib/docker/overlay2/3b6060eec3a1586a55a82fcdff198c231a72ae85ee6f6b4c6b13c556de95556d/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "old-k8s-version-745000",
"Source": "/var/lib/docker/volumes/old-k8s-version-745000/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "old-k8s-version-745000",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"22/tcp": {},
"2376/tcp": {},
"32443/tcp": {},
"5000/tcp": {},
"8443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "old-k8s-version-745000",
"name.minikube.sigs.k8s.io": "old-k8s-version-745000",
"role.minikube.sigs.k8s.io": ""
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "970e912762f28c45ce8dc481d3366240a812029a655c36560fed0e5d51dcb46d",
"SandboxKey": "/var/run/docker/netns/970e912762f2",
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "56464"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "56465"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "56466"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "56467"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "56463"
}
]
},
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"old-k8s-version-745000": {
"IPAMConfig": {
"IPv4Address": "192.168.67.2"
},
"Links": null,
"Aliases": [
"2b4f372aa246",
"old-k8s-version-745000"
],
"MacAddress": "02:42:c0:a8:43:02",
"NetworkID": "e9fab362389ee13cca953b7169efcc99796a0092a501ddc4284447becaba8d37",
"EndpointID": "3d8a93baad79ed1f421248ed1cca2ffc158043d55f4dc5df0ce630c93615549b",
"Gateway": "192.168.67.1",
"IPAddress": "192.168.67.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"DriverOpts": null,
"DNSNames": [
"old-k8s-version-745000",
"2b4f372aa246"
]
}
}
}
}
]
-- /stdout --
helpers_test.go:239: (dbg) Run: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-745000 -n old-k8s-version-745000
helpers_test.go:239: (dbg) Non-zero exit: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-745000 -n old-k8s-version-745000: exit status 6 (398.299732ms)
-- stdout --
Running
WARNING: Your kubectl is pointing to stale minikube-vm.
To fix the kubectl context, run `minikube update-context`
-- /stdout --
** stderr **
E0213 16:01:11.098509 23117 status.go:415] kubeconfig endpoint: extract IP: "old-k8s-version-745000" does not appear in /Users/jenkins/minikube-integration/18169-6320/kubeconfig
** /stderr **
helpers_test.go:239: status error: exit status 6 (may be ok)
helpers_test.go:241: "old-k8s-version-745000" host is not running, skipping log retrieval (state="Running\nWARNING: Your kubectl is pointing to stale minikube-vm.\nTo fix the kubectl context, run `minikube update-context`")
--- FAIL: TestStartStop/group/old-k8s-version/serial/FirstStart (259.68s)