=== RUN TestStartStop/group/old-k8s-version/serial/FirstStart
start_stop_delete_test.go:186: (dbg) Run: out/minikube-darwin-amd64 start -p old-k8s-version-079000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0
E0103 12:44:53.196300 11090 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/flannel-236000/client.crt: no such file or directory
E0103 12:44:58.421287 11090 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/functional-307000/client.crt: no such file or directory
E0103 12:45:03.436886 11090 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/flannel-236000/client.crt: no such file or directory
start_stop_delete_test.go:186: (dbg) Non-zero exit: out/minikube-darwin-amd64 start -p old-k8s-version-079000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0: exit status 109 (4m14.030898664s)
-- stdout --
* [old-k8s-version-079000] minikube v1.32.0 on Darwin 14.2
- MINIKUBE_LOCATION=17885
- KUBECONFIG=/Users/jenkins/minikube-integration/17885-10646/kubeconfig
- MINIKUBE_BIN=out/minikube-darwin-amd64
- MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
- MINIKUBE_HOME=/Users/jenkins/minikube-integration/17885-10646/.minikube
- MINIKUBE_FORCE_SYSTEMD=
* Using the docker driver based on user configuration
* Using Docker Desktop driver with root privileges
* Starting control plane node old-k8s-version-079000 in cluster old-k8s-version-079000
* Pulling base image v0.0.42-1703498848-17857 ...
* Creating docker container (CPUs=2, Memory=2200MB) ...
* Preparing Kubernetes v1.16.0 on Docker 24.0.7 ...
- Generating certificates and keys ...
- Booting up control plane ...
- Generating certificates and keys ...
- Booting up control plane ...
-- /stdout --
** stderr **
I0103 12:44:51.210715 27031 out.go:296] Setting OutFile to fd 1 ...
I0103 12:44:51.211008 27031 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0103 12:44:51.211015 27031 out.go:309] Setting ErrFile to fd 2...
I0103 12:44:51.211019 27031 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0103 12:44:51.211209 27031 root.go:338] Updating PATH: /Users/jenkins/minikube-integration/17885-10646/.minikube/bin
I0103 12:44:51.212765 27031 out.go:303] Setting JSON to false
I0103 12:44:51.236281 27031 start.go:128] hostinfo: {"hostname":"MacOS-Agent-2.local","uptime":8061,"bootTime":1704306630,"procs":458,"os":"darwin","platform":"darwin","platformFamily":"Standalone Workstation","platformVersion":"14.2","kernelVersion":"23.2.0","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"2965c349-98a5-5970-aaa9-9eedd3ae5959"}
W0103 12:44:51.236381 27031 start.go:136] gopshost.Virtualization returned error: not implemented yet
I0103 12:44:51.262714 27031 out.go:177] * [old-k8s-version-079000] minikube v1.32.0 on Darwin 14.2
I0103 12:44:51.335634 27031 out.go:177] - MINIKUBE_LOCATION=17885
I0103 12:44:51.303424 27031 notify.go:220] Checking for updates...
I0103 12:44:51.379408 27031 out.go:177] - KUBECONFIG=/Users/jenkins/minikube-integration/17885-10646/kubeconfig
I0103 12:44:51.422339 27031 out.go:177] - MINIKUBE_BIN=out/minikube-darwin-amd64
I0103 12:44:51.464486 27031 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0103 12:44:51.507429 27031 out.go:177] - MINIKUBE_HOME=/Users/jenkins/minikube-integration/17885-10646/.minikube
I0103 12:44:51.550412 27031 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0103 12:44:51.573071 27031 config.go:182] Loaded profile config "false-236000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.28.4
I0103 12:44:51.573213 27031 driver.go:392] Setting default libvirt URI to qemu:///system
I0103 12:44:51.637903 27031 docker.go:122] docker version: linux-24.0.7:Docker Desktop 4.26.0 (130397)
I0103 12:44:51.638062 27031 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0103 12:44:51.770534 27031 info.go:266] docker info: {ID:9dd12a49-41d2-44e8-aa64-4ab7fa99394e Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:58 OomKillDisable:false NGoroutines:68 SystemTime:2024-01-03 20:44:51.755873052 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:8 KernelVersion:6.5.11-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServe
rAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:12 MemTotal:6221275136 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:24.0.7 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f Expected:d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f} RuncCommit:{ID:v1.1.10-0-g18a0cb0 Expected:v1.1.10-0-g18a0cb0} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=unconf
ined name=cgroupns] ProductLicense: Warnings:[WARNING: daemon is not using the default seccomp profile] ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.0-desktop.2] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.23.3-desktop.2] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker Dev Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Mana
ges Docker extensions Vendor:Docker Inc. Version:v0.2.21] map[Name:feedback Path:/Users/jenkins/.docker/cli-plugins/docker-feedback SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-feedback] ShortDescription:Provide feedback, right in your terminal! Vendor:Docker Inc. Version:0.1] map[Name:init Path:/Users/jenkins/.docker/cli-plugins/docker-init SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-init] ShortDescription:Creates Docker-related starter files for your project Vendor:Docker Inc. Version:v0.1.0-beta.10] map[Name:sbom Path:/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM) for an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/d
ocker-scan] ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.26.0] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Docker Scout Vendor:Docker Inc. Version:v1.2.0]] Warnings:<nil>}}
I0103 12:44:51.794209 27031 out.go:177] * Using the docker driver based on user configuration
I0103 12:44:51.853354 27031 start.go:298] selected driver: docker
I0103 12:44:51.853378 27031 start.go:902] validating driver "docker" against <nil>
I0103 12:44:51.853396 27031 start.go:913] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0103 12:44:51.857572 27031 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0103 12:44:51.979958 27031 info.go:266] docker info: {ID:9dd12a49-41d2-44e8-aa64-4ab7fa99394e Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:58 OomKillDisable:false NGoroutines:68 SystemTime:2024-01-03 20:44:51.968997522 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:8 KernelVersion:6.5.11-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServe
rAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:12 MemTotal:6221275136 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:24.0.7 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f Expected:d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f} RuncCommit:{ID:v1.1.10-0-g18a0cb0 Expected:v1.1.10-0-g18a0cb0} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=unconf
ined name=cgroupns] ProductLicense: Warnings:[WARNING: daemon is not using the default seccomp profile] ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.0-desktop.2] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.23.3-desktop.2] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker Dev Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Mana
ges Docker extensions Vendor:Docker Inc. Version:v0.2.21] map[Name:feedback Path:/Users/jenkins/.docker/cli-plugins/docker-feedback SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-feedback] ShortDescription:Provide feedback, right in your terminal! Vendor:Docker Inc. Version:0.1] map[Name:init Path:/Users/jenkins/.docker/cli-plugins/docker-init SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-init] ShortDescription:Creates Docker-related starter files for your project Vendor:Docker Inc. Version:v0.1.0-beta.10] map[Name:sbom Path:/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM) for an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/d
ocker-scan] ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.26.0] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Docker Scout Vendor:Docker Inc. Version:v1.2.0]] Warnings:<nil>}}
I0103 12:44:51.980173 27031 start_flags.go:309] no existing cluster config was found, will generate one from the flags
I0103 12:44:51.980369 27031 start_flags.go:931] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0103 12:44:52.022381 27031 out.go:177] * Using Docker Desktop driver with root privileges
I0103 12:44:52.060775 27031 cni.go:84] Creating CNI manager for ""
I0103 12:44:52.060820 27031 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0103 12:44:52.060841 27031 start_flags.go:323] config:
{Name:old-k8s-version-079000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703498848-17857@sha256:81ae12a49915e4f02aa382dd3758a30a6649e1143c32b3d03309750104577c6c Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-079000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local Containe
rRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 AutoPauseInterval:1m0s GPUs:}
I0103 12:44:52.083344 27031 out.go:177] * Starting control plane node old-k8s-version-079000 in cluster old-k8s-version-079000
I0103 12:44:52.115237 27031 cache.go:121] Beginning downloading kic base image for docker with docker
I0103 12:44:52.152342 27031 out.go:177] * Pulling base image v0.0.42-1703498848-17857 ...
I0103 12:44:52.210628 27031 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0103 12:44:52.210723 27031 image.go:79] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703498848-17857@sha256:81ae12a49915e4f02aa382dd3758a30a6649e1143c32b3d03309750104577c6c in local docker daemon
I0103 12:44:52.210726 27031 preload.go:148] Found local preload: /Users/jenkins/minikube-integration/17885-10646/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4
I0103 12:44:52.210767 27031 cache.go:56] Caching tarball of preloaded images
I0103 12:44:52.210966 27031 preload.go:174] Found /Users/jenkins/minikube-integration/17885-10646/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0103 12:44:52.210989 27031 cache.go:59] Finished verifying existence of preloaded tar for v1.16.0 on docker
I0103 12:44:52.211189 27031 profile.go:148] Saving config to /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/config.json ...
I0103 12:44:52.212017 27031 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/config.json: {Name:mk56023a73a3d270c799e06d56151a29d2835afb Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0103 12:44:52.289648 27031 image.go:83] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703498848-17857@sha256:81ae12a49915e4f02aa382dd3758a30a6649e1143c32b3d03309750104577c6c in local docker daemon, skipping pull
I0103 12:44:52.289669 27031 cache.go:144] gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703498848-17857@sha256:81ae12a49915e4f02aa382dd3758a30a6649e1143c32b3d03309750104577c6c exists in daemon, skipping load
I0103 12:44:52.289693 27031 cache.go:194] Successfully downloaded all kic artifacts
I0103 12:44:52.289740 27031 start.go:365] acquiring machines lock for old-k8s-version-079000: {Name:mkefdae168ae5396c7edce5050a591938b306f62 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0103 12:44:52.289910 27031 start.go:369] acquired machines lock for "old-k8s-version-079000" in 152.126µs
I0103 12:44:52.289938 27031 start.go:93] Provisioning new machine with config: &{Name:old-k8s-version-079000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703498848-17857@sha256:81ae12a49915e4f02aa382dd3758a30a6649e1143c32b3d03309750104577c6c Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-079000 Namespace:default APIServer
Name:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:f
alse DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 AutoPauseInterval:1m0s GPUs:} &{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}
I0103 12:44:52.289993 27031 start.go:125] createHost starting for "" (driver="docker")
I0103 12:44:52.332299 27031 out.go:204] * Creating docker container (CPUs=2, Memory=2200MB) ...
I0103 12:44:52.332685 27031 start.go:159] libmachine.API.Create for "old-k8s-version-079000" (driver="docker")
I0103 12:44:52.332739 27031 client.go:168] LocalClient.Create starting
I0103 12:44:52.332982 27031 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/17885-10646/.minikube/certs/ca.pem
I0103 12:44:52.333068 27031 main.go:141] libmachine: Decoding PEM data...
I0103 12:44:52.333102 27031 main.go:141] libmachine: Parsing certificate...
I0103 12:44:52.333181 27031 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/17885-10646/.minikube/certs/cert.pem
I0103 12:44:52.333249 27031 main.go:141] libmachine: Decoding PEM data...
I0103 12:44:52.333265 27031 main.go:141] libmachine: Parsing certificate...
I0103 12:44:52.334095 27031 cli_runner.go:164] Run: docker network inspect old-k8s-version-079000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
W0103 12:44:52.395751 27031 cli_runner.go:211] docker network inspect old-k8s-version-079000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1
I0103 12:44:52.395850 27031 network_create.go:281] running [docker network inspect old-k8s-version-079000] to gather additional debugging logs...
I0103 12:44:52.395867 27031 cli_runner.go:164] Run: docker network inspect old-k8s-version-079000
W0103 12:44:52.447055 27031 cli_runner.go:211] docker network inspect old-k8s-version-079000 returned with exit code 1
I0103 12:44:52.447086 27031 network_create.go:284] error running [docker network inspect old-k8s-version-079000]: docker network inspect old-k8s-version-079000: exit status 1
stdout:
[]
stderr:
Error response from daemon: network old-k8s-version-079000 not found
I0103 12:44:52.447097 27031 network_create.go:286] output of [docker network inspect old-k8s-version-079000]: -- stdout --
[]
-- /stdout --
** stderr **
Error response from daemon: network old-k8s-version-079000 not found
** /stderr **
I0103 12:44:52.447287 27031 cli_runner.go:164] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0103 12:44:52.505717 27031 network.go:212] skipping subnet 192.168.49.0/24 that is reserved: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0103 12:44:52.507111 27031 network.go:212] skipping subnet 192.168.58.0/24 that is reserved: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0103 12:44:52.508569 27031 network.go:212] skipping subnet 192.168.67.0/24 that is reserved: &{IP:192.168.67.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.67.0/24 Gateway:192.168.67.1 ClientMin:192.168.67.2 ClientMax:192.168.67.254 Broadcast:192.168.67.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0103 12:44:52.508888 27031 network.go:209] using free private subnet 192.168.76.0/24: &{IP:192.168.76.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.76.0/24 Gateway:192.168.76.1 ClientMin:192.168.76.2 ClientMax:192.168.76.254 Broadcast:192.168.76.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc00226d730}
I0103 12:44:52.508903 27031 network_create.go:124] attempt to create docker network old-k8s-version-079000 192.168.76.0/24 with gateway 192.168.76.1 and MTU of 65535 ...
I0103 12:44:52.508968 27031 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.76.0/24 --gateway=192.168.76.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-079000 old-k8s-version-079000
I0103 12:44:52.608332 27031 network_create.go:108] docker network old-k8s-version-079000 192.168.76.0/24 created
I0103 12:44:52.608373 27031 kic.go:121] calculated static IP "192.168.76.2" for the "old-k8s-version-079000" container
I0103 12:44:52.608491 27031 cli_runner.go:164] Run: docker ps -a --format {{.Names}}
I0103 12:44:52.660391 27031 cli_runner.go:164] Run: docker volume create old-k8s-version-079000 --label name.minikube.sigs.k8s.io=old-k8s-version-079000 --label created_by.minikube.sigs.k8s.io=true
I0103 12:44:52.717794 27031 oci.go:103] Successfully created a docker volume old-k8s-version-079000
I0103 12:44:52.717898 27031 cli_runner.go:164] Run: docker run --rm --name old-k8s-version-079000-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-079000 --entrypoint /usr/bin/test -v old-k8s-version-079000:/var gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703498848-17857@sha256:81ae12a49915e4f02aa382dd3758a30a6649e1143c32b3d03309750104577c6c -d /var/lib
I0103 12:44:53.159395 27031 oci.go:107] Successfully prepared a docker volume old-k8s-version-079000
I0103 12:44:53.159440 27031 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0103 12:44:53.159455 27031 kic.go:194] Starting extracting preloaded images to volume ...
I0103 12:44:53.159566 27031 cli_runner.go:164] Run: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/17885-10646/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-079000:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703498848-17857@sha256:81ae12a49915e4f02aa382dd3758a30a6649e1143c32b3d03309750104577c6c -I lz4 -xf /preloaded.tar -C /extractDir
I0103 12:44:55.669710 27031 cli_runner.go:217] Completed: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/17885-10646/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-079000:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703498848-17857@sha256:81ae12a49915e4f02aa382dd3758a30a6649e1143c32b3d03309750104577c6c -I lz4 -xf /preloaded.tar -C /extractDir: (2.51009597s)
I0103 12:44:55.669743 27031 kic.go:203] duration metric: took 2.510319 seconds to extract preloaded images to volume
I0103 12:44:55.669895 27031 cli_runner.go:164] Run: docker info --format "'{{json .SecurityOptions}}'"
I0103 12:44:55.780738 27031 cli_runner.go:164] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname old-k8s-version-079000 --name old-k8s-version-079000 --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-079000 --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=old-k8s-version-079000 --network old-k8s-version-079000 --ip 192.168.76.2 --volume old-k8s-version-079000:/var --security-opt apparmor=unconfined --memory=2200mb --memory-swap=2200mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703498848-17857@sha256:81ae12a49915e4f02aa382dd3758a30a6649e1143c32b3d03309750104577c6c
I0103 12:44:56.099441 27031 cli_runner.go:164] Run: docker container inspect old-k8s-version-079000 --format={{.State.Running}}
I0103 12:44:56.167899 27031 cli_runner.go:164] Run: docker container inspect old-k8s-version-079000 --format={{.State.Status}}
I0103 12:44:56.257653 27031 cli_runner.go:164] Run: docker exec old-k8s-version-079000 stat /var/lib/dpkg/alternatives/iptables
I0103 12:44:56.415030 27031 oci.go:144] the created container "old-k8s-version-079000" has a running status.
I0103 12:44:56.415073 27031 kic.go:225] Creating ssh key for kic: /Users/jenkins/minikube-integration/17885-10646/.minikube/machines/old-k8s-version-079000/id_rsa...
I0103 12:44:56.582439 27031 kic_runner.go:191] docker (temp): /Users/jenkins/minikube-integration/17885-10646/.minikube/machines/old-k8s-version-079000/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0103 12:44:56.652915 27031 cli_runner.go:164] Run: docker container inspect old-k8s-version-079000 --format={{.State.Status}}
I0103 12:44:56.708628 27031 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0103 12:44:56.708650 27031 kic_runner.go:114] Args: [docker exec --privileged old-k8s-version-079000 chown docker:docker /home/docker/.ssh/authorized_keys]
I0103 12:44:56.796346 27031 cli_runner.go:164] Run: docker container inspect old-k8s-version-079000 --format={{.State.Status}}
I0103 12:44:56.847446 27031 machine.go:88] provisioning docker machine ...
I0103 12:44:56.847498 27031 ubuntu.go:169] provisioning hostname "old-k8s-version-079000"
I0103 12:44:56.847596 27031 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-079000
I0103 12:44:56.902389 27031 main.go:141] libmachine: Using SSH client type: native
I0103 12:44:56.902726 27031 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1406660] 0x1409340 <nil> [] 0s} 127.0.0.1 61413 <nil> <nil>}
I0103 12:44:56.902738 27031 main.go:141] libmachine: About to run SSH command:
sudo hostname old-k8s-version-079000 && echo "old-k8s-version-079000" | sudo tee /etc/hostname
I0103 12:44:57.033616 27031 main.go:141] libmachine: SSH cmd err, output: <nil>: old-k8s-version-079000
I0103 12:44:57.033721 27031 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-079000
I0103 12:44:57.086739 27031 main.go:141] libmachine: Using SSH client type: native
I0103 12:44:57.087041 27031 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1406660] 0x1409340 <nil> [] 0s} 127.0.0.1 61413 <nil> <nil>}
I0103 12:44:57.087055 27031 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sold-k8s-version-079000' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 old-k8s-version-079000/g' /etc/hosts;
else
echo '127.0.1.1 old-k8s-version-079000' | sudo tee -a /etc/hosts;
fi
fi
I0103 12:44:57.206904 27031 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0103 12:44:57.206928 27031 ubuntu.go:175] set auth options {CertDir:/Users/jenkins/minikube-integration/17885-10646/.minikube CaCertPath:/Users/jenkins/minikube-integration/17885-10646/.minikube/certs/ca.pem CaPrivateKeyPath:/Users/jenkins/minikube-integration/17885-10646/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/Users/jenkins/minikube-integration/17885-10646/.minikube/machines/server.pem ServerKeyPath:/Users/jenkins/minikube-integration/17885-10646/.minikube/machines/server-key.pem ClientKeyPath:/Users/jenkins/minikube-integration/17885-10646/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/Users/jenkins/minikube-integration/17885-10646/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/Users/jenkins/minikube-integration/17885-10646/.minikube}
I0103 12:44:57.206946 27031 ubuntu.go:177] setting up certificates
I0103 12:44:57.206958 27031 provision.go:83] configureAuth start
I0103 12:44:57.207034 27031 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-079000
I0103 12:44:57.258878 27031 provision.go:138] copyHostCerts
I0103 12:44:57.258983 27031 exec_runner.go:144] found /Users/jenkins/minikube-integration/17885-10646/.minikube/key.pem, removing ...
I0103 12:44:57.258994 27031 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/17885-10646/.minikube/key.pem
I0103 12:44:57.259696 27031 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/17885-10646/.minikube/certs/key.pem --> /Users/jenkins/minikube-integration/17885-10646/.minikube/key.pem (1679 bytes)
I0103 12:44:57.259928 27031 exec_runner.go:144] found /Users/jenkins/minikube-integration/17885-10646/.minikube/ca.pem, removing ...
I0103 12:44:57.259934 27031 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/17885-10646/.minikube/ca.pem
I0103 12:44:57.260029 27031 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/17885-10646/.minikube/certs/ca.pem --> /Users/jenkins/minikube-integration/17885-10646/.minikube/ca.pem (1078 bytes)
I0103 12:44:57.298053 27031 exec_runner.go:144] found /Users/jenkins/minikube-integration/17885-10646/.minikube/cert.pem, removing ...
I0103 12:44:57.298067 27031 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/17885-10646/.minikube/cert.pem
I0103 12:44:57.298199 27031 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/17885-10646/.minikube/certs/cert.pem --> /Users/jenkins/minikube-integration/17885-10646/.minikube/cert.pem (1123 bytes)
I0103 12:44:57.298426 27031 provision.go:112] generating server cert: /Users/jenkins/minikube-integration/17885-10646/.minikube/machines/server.pem ca-key=/Users/jenkins/minikube-integration/17885-10646/.minikube/certs/ca.pem private-key=/Users/jenkins/minikube-integration/17885-10646/.minikube/certs/ca-key.pem org=jenkins.old-k8s-version-079000 san=[192.168.76.2 127.0.0.1 localhost 127.0.0.1 minikube old-k8s-version-079000]
I0103 12:44:57.358778 27031 provision.go:172] copyRemoteCerts
I0103 12:44:57.358842 27031 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0103 12:44:57.358898 27031 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-079000
I0103 12:44:57.410549 27031 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:61413 SSHKeyPath:/Users/jenkins/minikube-integration/17885-10646/.minikube/machines/old-k8s-version-079000/id_rsa Username:docker}
I0103 12:44:57.496355 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1078 bytes)
I0103 12:44:57.516611 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/machines/server.pem --> /etc/docker/server.pem (1241 bytes)
I0103 12:44:57.537839 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1675 bytes)
I0103 12:44:57.558787 27031 provision.go:86] duration metric: configureAuth took 351.817535ms
I0103 12:44:57.558807 27031 ubuntu.go:193] setting minikube options for container-runtime
I0103 12:44:57.558948 27031 config.go:182] Loaded profile config "old-k8s-version-079000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.16.0
I0103 12:44:57.559030 27031 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-079000
I0103 12:44:57.611666 27031 main.go:141] libmachine: Using SSH client type: native
I0103 12:44:57.611965 27031 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1406660] 0x1409340 <nil> [] 0s} 127.0.0.1 61413 <nil> <nil>}
I0103 12:44:57.611978 27031 main.go:141] libmachine: About to run SSH command:
df --output=fstype / | tail -n 1
I0103 12:44:57.731560 27031 main.go:141] libmachine: SSH cmd err, output: <nil>: overlay
I0103 12:44:57.731576 27031 ubuntu.go:71] root file system type: overlay
I0103 12:44:57.731658 27031 provision.go:309] Updating docker unit: /lib/systemd/system/docker.service ...
I0103 12:44:57.731737 27031 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-079000
I0103 12:44:57.789104 27031 main.go:141] libmachine: Using SSH client type: native
I0103 12:44:57.789394 27031 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1406660] 0x1409340 <nil> [] 0s} 127.0.0.1 61413 <nil> <nil>}
I0103 12:44:57.789449 27031 main.go:141] libmachine: About to run SSH command:
sudo mkdir -p /lib/systemd/system && printf %s "[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP \$MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service.new
I0103 12:44:57.921269 27031 main.go:141] libmachine: SSH cmd err, output: <nil>: [Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
I0103 12:44:57.921377 27031 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-079000
I0103 12:44:57.983252 27031 main.go:141] libmachine: Using SSH client type: native
I0103 12:44:57.983708 27031 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1406660] 0x1409340 <nil> [] 0s} 127.0.0.1 61413 <nil> <nil>}
I0103 12:44:57.983740 27031 main.go:141] libmachine: About to run SSH command:
sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
I0103 12:44:58.669585 27031 main.go:141] libmachine: SSH cmd err, output: <nil>: --- /lib/systemd/system/docker.service 2023-10-26 09:06:22.000000000 +0000
+++ /lib/systemd/system/docker.service.new 2024-01-03 20:44:57.918598757 +0000
@@ -1,30 +1,32 @@
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
-After=network-online.target docker.socket firewalld.service containerd.service time-set.target
-Wants=network-online.target containerd.service
+BindsTo=containerd.service
+After=network-online.target firewalld.service containerd.service
+Wants=network-online.target
Requires=docker.socket
+StartLimitBurst=3
+StartLimitIntervalSec=60
[Service]
Type=notify
-# the default is not to use systemd for cgroups because the delegate issues still
-# exists and systemd currently does not support the cgroup feature set required
-# for containers run by docker
-ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
-ExecReload=/bin/kill -s HUP $MAINPID
-TimeoutStartSec=0
-RestartSec=2
-Restart=always
+Restart=on-failure
-# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
-# Both the old, and new location are accepted by systemd 229 and up, so using the old location
-# to make them work for either version of systemd.
-StartLimitBurst=3
-# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
-# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
-# this option work for either version of systemd.
-StartLimitInterval=60s
+
+# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
+# The base configuration already specifies an 'ExecStart=...' command. The first directive
+# here is to clear out that command inherited from the base configuration. Without this,
+# the command from the base configuration and the command specified here are treated as
+# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
+# will catch this invalid input and refuse to start the service with an error like:
+# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
+
+# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
+# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
+ExecStart=
+ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
+ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
@@ -32,16 +34,16 @@
LimitNPROC=infinity
LimitCORE=infinity
-# Comment TasksMax if your systemd version does not support it.
-# Only systemd 226 and above support this option.
+# Uncomment TasksMax if your systemd version supports it.
+# Only systemd 226 and above support this version.
TasksMax=infinity
+TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
-OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
I0103 12:44:58.669608 27031 machine.go:91] provisioned docker machine in 1.82215482s
I0103 12:44:58.669623 27031 client.go:171] LocalClient.Create took 6.336958006s
I0103 12:44:58.669639 27031 start.go:167] duration metric: libmachine.API.Create for "old-k8s-version-079000" took 6.337037993s
I0103 12:44:58.669647 27031 start.go:300] post-start starting for "old-k8s-version-079000" (driver="docker")
I0103 12:44:58.669659 27031 start.go:329] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0103 12:44:58.669732 27031 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0103 12:44:58.669800 27031 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-079000
I0103 12:44:58.722674 27031 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:61413 SSHKeyPath:/Users/jenkins/minikube-integration/17885-10646/.minikube/machines/old-k8s-version-079000/id_rsa Username:docker}
I0103 12:44:58.809487 27031 ssh_runner.go:195] Run: cat /etc/os-release
I0103 12:44:58.813382 27031 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0103 12:44:58.813409 27031 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0103 12:44:58.813417 27031 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0103 12:44:58.813423 27031 info.go:137] Remote host: Ubuntu 22.04.3 LTS
I0103 12:44:58.813434 27031 filesync.go:126] Scanning /Users/jenkins/minikube-integration/17885-10646/.minikube/addons for local assets ...
I0103 12:44:58.813540 27031 filesync.go:126] Scanning /Users/jenkins/minikube-integration/17885-10646/.minikube/files for local assets ...
I0103 12:44:58.813724 27031 filesync.go:149] local asset: /Users/jenkins/minikube-integration/17885-10646/.minikube/files/etc/ssl/certs/110902.pem -> 110902.pem in /etc/ssl/certs
I0103 12:44:58.813926 27031 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0103 12:44:58.822139 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/files/etc/ssl/certs/110902.pem --> /etc/ssl/certs/110902.pem (1708 bytes)
I0103 12:44:58.842884 27031 start.go:303] post-start completed in 173.229969ms
I0103 12:44:58.843475 27031 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-079000
I0103 12:44:58.896088 27031 profile.go:148] Saving config to /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/config.json ...
I0103 12:44:58.896543 27031 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0103 12:44:58.896603 27031 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-079000
I0103 12:44:58.951930 27031 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:61413 SSHKeyPath:/Users/jenkins/minikube-integration/17885-10646/.minikube/machines/old-k8s-version-079000/id_rsa Username:docker}
I0103 12:44:59.035364 27031 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0103 12:44:59.040560 27031 start.go:128] duration metric: createHost completed in 6.750636066s
I0103 12:44:59.040581 27031 start.go:83] releasing machines lock for "old-k8s-version-079000", held for 6.750748089s
I0103 12:44:59.040671 27031 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-079000
I0103 12:44:59.102025 27031 ssh_runner.go:195] Run: cat /version.json
I0103 12:44:59.102050 27031 ssh_runner.go:195] Run: curl -sS -m 2 https://registry.k8s.io/
I0103 12:44:59.102102 27031 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-079000
I0103 12:44:59.102116 27031 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-079000
I0103 12:44:59.157595 27031 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:61413 SSHKeyPath:/Users/jenkins/minikube-integration/17885-10646/.minikube/machines/old-k8s-version-079000/id_rsa Username:docker}
I0103 12:44:59.157790 27031 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:61413 SSHKeyPath:/Users/jenkins/minikube-integration/17885-10646/.minikube/machines/old-k8s-version-079000/id_rsa Username:docker}
I0103 12:44:59.351982 27031 ssh_runner.go:195] Run: systemctl --version
I0103 12:44:59.356875 27031 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
I0103 12:44:59.362208 27031 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *loopback.conf* -not -name *.mk_disabled -exec sh -c "grep -q loopback {} && ( grep -q name {} || sudo sed -i '/"type": "loopback"/i \ \ \ \ "name": "loopback",' {} ) && sudo sed -i 's|"cniVersion": ".*"|"cniVersion": "1.0.0"|g' {}" ;
I0103 12:44:59.384999 27031 cni.go:230] loopback cni configuration patched: "/etc/cni/net.d/*loopback.conf*" found
I0103 12:44:59.385112 27031 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *bridge* -not -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e '/"dst": ".*:.*"/d' -e 's|^(.*)"dst": (.*)[,*]$|\1"dst": \2|g' -e '/"subnet": ".*:.*"/d' -e 's|^(.*)"subnet": ".*"(.*)[,*]$|\1"subnet": "10.244.0.0/16"\2|g' {}" ;
I0103 12:44:59.400995 27031 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e 's|^(.*)"subnet": ".*"(.*)$|\1"subnet": "10.244.0.0/16"\2|g' -e 's|^(.*)"gateway": ".*"(.*)$|\1"gateway": "10.244.0.1"\2|g' {}" ;
I0103 12:44:59.417063 27031 cni.go:308] configured [/etc/cni/net.d/100-crio-bridge.conf, /etc/cni/net.d/87-podman-bridge.conflist] bridge cni config(s)
I0103 12:44:59.417080 27031 start.go:475] detecting cgroup driver to use...
I0103 12:44:59.417102 27031 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0103 12:44:59.417230 27031 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0103 12:44:59.433086 27031 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.1"|' /etc/containerd/config.toml"
I0103 12:44:59.443203 27031 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I0103 12:44:59.453900 27031 containerd.go:145] configuring containerd to use "cgroupfs" as cgroup driver...
I0103 12:44:59.453970 27031 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = false|g' /etc/containerd/config.toml"
I0103 12:44:59.464608 27031 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0103 12:44:59.474425 27031 ssh_runner.go:195] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I0103 12:44:59.484813 27031 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0103 12:44:59.494479 27031 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0103 12:44:59.503553 27031 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I0103 12:44:59.513189 27031 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0103 12:44:59.521523 27031 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0103 12:44:59.530390 27031 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0103 12:44:59.578998 27031 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0103 12:44:59.656612 27031 start.go:475] detecting cgroup driver to use...
I0103 12:44:59.656631 27031 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0103 12:44:59.656697 27031 ssh_runner.go:195] Run: sudo systemctl cat docker.service
I0103 12:44:59.675097 27031 cruntime.go:279] skipping containerd shutdown because we are bound to it
I0103 12:44:59.675227 27031 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0103 12:44:59.686857 27031 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///var/run/dockershim.sock
" | sudo tee /etc/crictl.yaml"
I0103 12:44:59.703476 27031 ssh_runner.go:195] Run: which cri-dockerd
I0103 12:44:59.708371 27031 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/cri-docker.service.d
I0103 12:44:59.717518 27031 ssh_runner.go:362] scp memory --> /etc/systemd/system/cri-docker.service.d/10-cni.conf (189 bytes)
I0103 12:44:59.735531 27031 ssh_runner.go:195] Run: sudo systemctl unmask docker.service
I0103 12:44:59.822020 27031 ssh_runner.go:195] Run: sudo systemctl enable docker.socket
I0103 12:44:59.899083 27031 docker.go:560] configuring docker to use "cgroupfs" as cgroup driver...
I0103 12:44:59.899183 27031 ssh_runner.go:362] scp memory --> /etc/docker/daemon.json (130 bytes)
I0103 12:44:59.916245 27031 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0103 12:45:00.005143 27031 ssh_runner.go:195] Run: sudo systemctl restart docker
I0103 12:45:00.246955 27031 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0103 12:45:00.271083 27031 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0103 12:45:00.356051 27031 out.go:204] * Preparing Kubernetes v1.16.0 on Docker 24.0.7 ...
I0103 12:45:00.356335 27031 cli_runner.go:164] Run: docker exec -t old-k8s-version-079000 dig +short host.docker.internal
I0103 12:45:00.470868 27031 network.go:96] got host ip for mount in container by digging dns: 192.168.65.254
I0103 12:45:00.470955 27031 ssh_runner.go:195] Run: grep 192.168.65.254 host.minikube.internal$ /etc/hosts
I0103 12:45:00.475831 27031 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.65.254 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0103 12:45:00.486892 27031 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "8443/tcp") 0).HostPort}}'" old-k8s-version-079000
I0103 12:45:00.539385 27031 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0103 12:45:00.539475 27031 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0103 12:45:00.559279 27031 docker.go:671] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0103 12:45:00.559296 27031 docker.go:677] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0103 12:45:00.559366 27031 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0103 12:45:00.567848 27031 ssh_runner.go:195] Run: which lz4
I0103 12:45:00.572211 27031 ssh_runner.go:195] Run: stat -c "%s %y" /preloaded.tar.lz4
I0103 12:45:00.576691 27031 ssh_runner.go:352] existence check for /preloaded.tar.lz4: stat -c "%s %y" /preloaded.tar.lz4: Process exited with status 1
stdout:
stderr:
stat: cannot statx '/preloaded.tar.lz4': No such file or directory
I0103 12:45:00.576723 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 --> /preloaded.tar.lz4 (369789069 bytes)
I0103 12:45:05.792466 27031 docker.go:635] Took 5.220369 seconds to copy over tarball
I0103 12:45:05.792549 27031 ssh_runner.go:195] Run: sudo tar -I lz4 -C /var -xf /preloaded.tar.lz4
I0103 12:45:07.369497 27031 ssh_runner.go:235] Completed: sudo tar -I lz4 -C /var -xf /preloaded.tar.lz4: (1.576952841s)
I0103 12:45:07.369511 27031 ssh_runner.go:146] rm: /preloaded.tar.lz4
I0103 12:45:07.408767 27031 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0103 12:45:07.417975 27031 ssh_runner.go:362] scp memory --> /var/lib/docker/image/overlay2/repositories.json (2499 bytes)
I0103 12:45:07.433868 27031 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0103 12:45:07.490282 27031 ssh_runner.go:195] Run: sudo systemctl restart docker
I0103 12:45:08.173611 27031 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0103 12:45:08.193391 27031 docker.go:671] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0103 12:45:08.193406 27031 docker.go:677] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0103 12:45:08.193417 27031 cache_images.go:88] LoadImages start: [registry.k8s.io/kube-apiserver:v1.16.0 registry.k8s.io/kube-controller-manager:v1.16.0 registry.k8s.io/kube-scheduler:v1.16.0 registry.k8s.io/kube-proxy:v1.16.0 registry.k8s.io/pause:3.1 registry.k8s.io/etcd:3.3.15-0 registry.k8s.io/coredns:1.6.2 gcr.io/k8s-minikube/storage-provisioner:v5]
I0103 12:45:08.199771 27031 image.go:134] retrieving image: registry.k8s.io/etcd:3.3.15-0
I0103 12:45:08.201209 27031 image.go:134] retrieving image: registry.k8s.io/kube-apiserver:v1.16.0
I0103 12:45:08.202687 27031 image.go:134] retrieving image: registry.k8s.io/pause:3.1
I0103 12:45:08.202732 27031 image.go:134] retrieving image: registry.k8s.io/coredns:1.6.2
I0103 12:45:08.203037 27031 image.go:134] retrieving image: registry.k8s.io/kube-proxy:v1.16.0
I0103 12:45:08.203747 27031 image.go:134] retrieving image: registry.k8s.io/kube-scheduler:v1.16.0
I0103 12:45:08.204999 27031 image.go:134] retrieving image: gcr.io/k8s-minikube/storage-provisioner:v5
I0103 12:45:08.205711 27031 image.go:134] retrieving image: registry.k8s.io/kube-controller-manager:v1.16.0
I0103 12:45:08.207026 27031 image.go:177] daemon lookup for registry.k8s.io/kube-apiserver:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-apiserver:v1.16.0
I0103 12:45:08.208927 27031 image.go:177] daemon lookup for registry.k8s.io/pause:3.1: Error response from daemon: No such image: registry.k8s.io/pause:3.1
I0103 12:45:08.208939 27031 image.go:177] daemon lookup for registry.k8s.io/coredns:1.6.2: Error response from daemon: No such image: registry.k8s.io/coredns:1.6.2
I0103 12:45:08.209325 27031 image.go:177] daemon lookup for registry.k8s.io/etcd:3.3.15-0: Error response from daemon: No such image: registry.k8s.io/etcd:3.3.15-0
I0103 12:45:08.210717 27031 image.go:177] daemon lookup for registry.k8s.io/kube-proxy:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-proxy:v1.16.0
I0103 12:45:08.210789 27031 image.go:177] daemon lookup for registry.k8s.io/kube-scheduler:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-scheduler:v1.16.0
I0103 12:45:08.211430 27031 image.go:177] daemon lookup for gcr.io/k8s-minikube/storage-provisioner:v5: Error response from daemon: No such image: gcr.io/k8s-minikube/storage-provisioner:v5
I0103 12:45:08.211652 27031 image.go:177] daemon lookup for registry.k8s.io/kube-controller-manager:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-controller-manager:v1.16.0
I0103 12:45:08.634613 27031 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-apiserver:v1.16.0
I0103 12:45:08.640509 27031 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/pause:3.1
I0103 12:45:08.651550 27031 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/coredns:1.6.2
I0103 12:45:08.658137 27031 cache_images.go:116] "registry.k8s.io/kube-apiserver:v1.16.0" needs transfer: "registry.k8s.io/kube-apiserver:v1.16.0" does not exist at hash "b305571ca60a5a7818bda47da122683d75e8a1907475681ee8b1efbd06bff12e" in container runtime
I0103 12:45:08.658180 27031 docker.go:323] Removing image: registry.k8s.io/kube-apiserver:v1.16.0
I0103 12:45:08.658245 27031 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-apiserver:v1.16.0
I0103 12:45:08.665482 27031 cache_images.go:116] "registry.k8s.io/pause:3.1" needs transfer: "registry.k8s.io/pause:3.1" does not exist at hash "da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e" in container runtime
I0103 12:45:08.665516 27031 docker.go:323] Removing image: registry.k8s.io/pause:3.1
I0103 12:45:08.665572 27031 ssh_runner.go:195] Run: docker rmi registry.k8s.io/pause:3.1
I0103 12:45:08.678331 27031 cache_images.go:116] "registry.k8s.io/coredns:1.6.2" needs transfer: "registry.k8s.io/coredns:1.6.2" does not exist at hash "bf261d157914477ee1a5969d28ec687f3fbfc9fb5a664b22df78e57023b0e03b" in container runtime
I0103 12:45:08.678378 27031 docker.go:323] Removing image: registry.k8s.io/coredns:1.6.2
I0103 12:45:08.678459 27031 ssh_runner.go:195] Run: docker rmi registry.k8s.io/coredns:1.6.2
I0103 12:45:08.685721 27031 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17885-10646/.minikube/cache/images/amd64/registry.k8s.io/kube-apiserver_v1.16.0
I0103 12:45:08.689788 27031 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/etcd:3.3.15-0
I0103 12:45:08.693304 27031 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17885-10646/.minikube/cache/images/amd64/registry.k8s.io/pause_3.1
I0103 12:45:08.705124 27031 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17885-10646/.minikube/cache/images/amd64/registry.k8s.io/coredns_1.6.2
I0103 12:45:08.713545 27031 cache_images.go:116] "registry.k8s.io/etcd:3.3.15-0" needs transfer: "registry.k8s.io/etcd:3.3.15-0" does not exist at hash "b2756210eeabf84f3221da9959e9483f3919dc2aaab4cd45e7cd072fcbde27ed" in container runtime
I0103 12:45:08.713576 27031 docker.go:323] Removing image: registry.k8s.io/etcd:3.3.15-0
I0103 12:45:08.713642 27031 ssh_runner.go:195] Run: docker rmi registry.k8s.io/etcd:3.3.15-0
I0103 12:45:08.715260 27031 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-proxy:v1.16.0
I0103 12:45:08.736444 27031 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17885-10646/.minikube/cache/images/amd64/registry.k8s.io/etcd_3.3.15-0
I0103 12:45:08.737428 27031 cache_images.go:116] "registry.k8s.io/kube-proxy:v1.16.0" needs transfer: "registry.k8s.io/kube-proxy:v1.16.0" does not exist at hash "c21b0c7400f988db4777858edd13b6d3930d62d7ccf026d2415485a52037f384" in container runtime
I0103 12:45:08.737451 27031 docker.go:323] Removing image: registry.k8s.io/kube-proxy:v1.16.0
I0103 12:45:08.737508 27031 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-proxy:v1.16.0
I0103 12:45:08.753860 27031 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17885-10646/.minikube/cache/images/amd64/registry.k8s.io/kube-proxy_v1.16.0
I0103 12:45:08.799575 27031 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-scheduler:v1.16.0
I0103 12:45:08.819941 27031 cache_images.go:116] "registry.k8s.io/kube-scheduler:v1.16.0" needs transfer: "registry.k8s.io/kube-scheduler:v1.16.0" does not exist at hash "301ddc62b80b16315d3c2653cf3888370394277afb3187614cfa20edc352ca0a" in container runtime
I0103 12:45:08.819974 27031 docker.go:323] Removing image: registry.k8s.io/kube-scheduler:v1.16.0
I0103 12:45:08.820045 27031 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-scheduler:v1.16.0
I0103 12:45:08.838418 27031 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17885-10646/.minikube/cache/images/amd64/registry.k8s.io/kube-scheduler_v1.16.0
I0103 12:45:08.914538 27031 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-controller-manager:v1.16.0
I0103 12:45:08.935245 27031 cache_images.go:116] "registry.k8s.io/kube-controller-manager:v1.16.0" needs transfer: "registry.k8s.io/kube-controller-manager:v1.16.0" does not exist at hash "06a629a7e51cdcc81a5ed6a3e6650348312f20c954ac52ee489a023628ec9c7d" in container runtime
I0103 12:45:08.935270 27031 docker.go:323] Removing image: registry.k8s.io/kube-controller-manager:v1.16.0
I0103 12:45:08.935357 27031 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-controller-manager:v1.16.0
I0103 12:45:08.957272 27031 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17885-10646/.minikube/cache/images/amd64/registry.k8s.io/kube-controller-manager_v1.16.0
I0103 12:45:09.194744 27031 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} gcr.io/k8s-minikube/storage-provisioner:v5
I0103 12:45:09.214335 27031 cache_images.go:92] LoadImages completed in 1.020914653s
W0103 12:45:09.214395 27031 out.go:239] X Unable to load cached images: loading cached images: stat /Users/jenkins/minikube-integration/17885-10646/.minikube/cache/images/amd64/registry.k8s.io/kube-apiserver_v1.16.0: no such file or directory
X Unable to load cached images: loading cached images: stat /Users/jenkins/minikube-integration/17885-10646/.minikube/cache/images/amd64/registry.k8s.io/kube-apiserver_v1.16.0: no such file or directory
I0103 12:45:09.214481 27031 ssh_runner.go:195] Run: docker info --format {{.CgroupDriver}}
I0103 12:45:09.264328 27031 cni.go:84] Creating CNI manager for ""
I0103 12:45:09.264346 27031 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0103 12:45:09.264359 27031 kubeadm.go:87] Using pod CIDR: 10.244.0.0/16
I0103 12:45:09.264376 27031 kubeadm.go:176] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.76.2 APIServerPort:8443 KubernetesVersion:v1.16.0 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:old-k8s-version-079000 NodeName:old-k8s-version-079000 DNSDomain:cluster.local CRISocket:/var/run/dockershim.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.76.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.76.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticP
odPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[hairpinMode:hairpin-veth runtimeRequestTimeout:15m] PrependCriSocketUnix:false}
I0103 12:45:09.264479 27031 kubeadm.go:181] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.76.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: "old-k8s-version-079000"
kubeletExtraArgs:
node-ip: 192.168.76.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.76.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: old-k8s-version-079000
controlPlaneEndpoint: control-plane.minikube.internal:8443
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
listen-metrics-urls: http://127.0.0.1:2381,http://192.168.76.2:2381
kubernetesVersion: v1.16.0
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0103 12:45:09.264529 27031 kubeadm.go:976] kubelet [Unit]
Wants=docker.socket
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.16.0/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=old-k8s-version-079000 --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.76.2
[Install]
config:
{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-079000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:}
I0103 12:45:09.264589 27031 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.16.0
I0103 12:45:09.273272 27031 binaries.go:44] Found k8s binaries, skipping transfer
I0103 12:45:09.273333 27031 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0103 12:45:09.281649 27031 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (348 bytes)
I0103 12:45:09.297176 27031 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0103 12:45:09.312525 27031 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2174 bytes)
I0103 12:45:09.328332 27031 ssh_runner.go:195] Run: grep 192.168.76.2 control-plane.minikube.internal$ /etc/hosts
I0103 12:45:09.332604 27031 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.76.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0103 12:45:09.343152 27031 certs.go:56] Setting up /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000 for IP: 192.168.76.2
I0103 12:45:09.343173 27031 certs.go:190] acquiring lock for shared ca certs: {Name:mk5a30c05f18415c794a1ae2617714fd3a6ba516 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0103 12:45:09.343364 27031 certs.go:199] skipping minikubeCA CA generation: /Users/jenkins/minikube-integration/17885-10646/.minikube/ca.key
I0103 12:45:09.343444 27031 certs.go:199] skipping proxyClientCA CA generation: /Users/jenkins/minikube-integration/17885-10646/.minikube/proxy-client-ca.key
I0103 12:45:09.343489 27031 certs.go:319] generating minikube-user signed cert: /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/client.key
I0103 12:45:09.343500 27031 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/client.crt with IP's: []
I0103 12:45:09.583325 27031 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/client.crt ...
I0103 12:45:09.583343 27031 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/client.crt: {Name:mkee1890013c8c4e294fc0f434bcfa1f989bc2c9 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0103 12:45:09.583694 27031 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/client.key ...
I0103 12:45:09.583703 27031 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/client.key: {Name:mkf9921ed5e670786ee16e432459c14235b8e3f7 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0103 12:45:09.583941 27031 certs.go:319] generating minikube signed cert: /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/apiserver.key.31bdca25
I0103 12:45:09.583957 27031 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/apiserver.crt.31bdca25 with IP's: [192.168.76.2 10.96.0.1 127.0.0.1 10.0.0.1]
I0103 12:45:09.875191 27031 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/apiserver.crt.31bdca25 ...
I0103 12:45:09.875212 27031 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/apiserver.crt.31bdca25: {Name:mk5e1479834f0f5f73263414d178142e4b1af2e2 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0103 12:45:09.875505 27031 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/apiserver.key.31bdca25 ...
I0103 12:45:09.875515 27031 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/apiserver.key.31bdca25: {Name:mk376cc06d9fa69620816b1e663255737bf8d9ff Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0103 12:45:09.875728 27031 certs.go:337] copying /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/apiserver.crt.31bdca25 -> /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/apiserver.crt
I0103 12:45:09.875920 27031 certs.go:341] copying /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/apiserver.key.31bdca25 -> /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/apiserver.key
I0103 12:45:09.876091 27031 certs.go:319] generating aggregator signed cert: /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/proxy-client.key
I0103 12:45:09.876105 27031 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/proxy-client.crt with IP's: []
I0103 12:45:09.963129 27031 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/proxy-client.crt ...
I0103 12:45:09.963144 27031 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/proxy-client.crt: {Name:mk85e3216a0acaec8eaad264f9849327530b6f8e Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0103 12:45:09.963468 27031 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/proxy-client.key ...
I0103 12:45:09.963477 27031 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/proxy-client.key: {Name:mke0886fb1fa6ecbe4b50b660e837c18f5841a3e Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0103 12:45:09.963921 27031 certs.go:437] found cert: /Users/jenkins/minikube-integration/17885-10646/.minikube/certs/Users/jenkins/minikube-integration/17885-10646/.minikube/certs/11090.pem (1338 bytes)
W0103 12:45:09.964005 27031 certs.go:433] ignoring /Users/jenkins/minikube-integration/17885-10646/.minikube/certs/Users/jenkins/minikube-integration/17885-10646/.minikube/certs/11090_empty.pem, impossibly tiny 0 bytes
I0103 12:45:09.964019 27031 certs.go:437] found cert: /Users/jenkins/minikube-integration/17885-10646/.minikube/certs/Users/jenkins/minikube-integration/17885-10646/.minikube/certs/ca-key.pem (1675 bytes)
I0103 12:45:09.964050 27031 certs.go:437] found cert: /Users/jenkins/minikube-integration/17885-10646/.minikube/certs/Users/jenkins/minikube-integration/17885-10646/.minikube/certs/ca.pem (1078 bytes)
I0103 12:45:09.964080 27031 certs.go:437] found cert: /Users/jenkins/minikube-integration/17885-10646/.minikube/certs/Users/jenkins/minikube-integration/17885-10646/.minikube/certs/cert.pem (1123 bytes)
I0103 12:45:09.964110 27031 certs.go:437] found cert: /Users/jenkins/minikube-integration/17885-10646/.minikube/certs/Users/jenkins/minikube-integration/17885-10646/.minikube/certs/key.pem (1679 bytes)
I0103 12:45:09.964178 27031 certs.go:437] found cert: /Users/jenkins/minikube-integration/17885-10646/.minikube/files/etc/ssl/certs/Users/jenkins/minikube-integration/17885-10646/.minikube/files/etc/ssl/certs/110902.pem (1708 bytes)
I0103 12:45:09.964681 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1399 bytes)
I0103 12:45:09.985868 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1675 bytes)
I0103 12:45:10.006828 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0103 12:45:10.027567 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/profiles/old-k8s-version-079000/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1679 bytes)
I0103 12:45:10.048305 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0103 12:45:10.069978 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1675 bytes)
I0103 12:45:10.091588 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0103 12:45:10.112270 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1675 bytes)
I0103 12:45:10.133219 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/certs/11090.pem --> /usr/share/ca-certificates/11090.pem (1338 bytes)
I0103 12:45:10.154158 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/files/etc/ssl/certs/110902.pem --> /usr/share/ca-certificates/110902.pem (1708 bytes)
I0103 12:45:10.175256 27031 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17885-10646/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0103 12:45:10.196415 27031 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0103 12:45:10.212160 27031 ssh_runner.go:195] Run: openssl version
I0103 12:45:10.217858 27031 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/110902.pem && ln -fs /usr/share/ca-certificates/110902.pem /etc/ssl/certs/110902.pem"
I0103 12:45:10.227319 27031 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/110902.pem
I0103 12:45:10.231575 27031 certs.go:480] hashing: -rw-r--r-- 1 root root 1708 Jan 3 19:57 /usr/share/ca-certificates/110902.pem
I0103 12:45:10.231626 27031 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/110902.pem
I0103 12:45:10.238375 27031 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/110902.pem /etc/ssl/certs/3ec20f2e.0"
I0103 12:45:10.247586 27031 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0103 12:45:10.256488 27031 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0103 12:45:10.260775 27031 certs.go:480] hashing: -rw-r--r-- 1 root root 1111 Jan 3 19:52 /usr/share/ca-certificates/minikubeCA.pem
I0103 12:45:10.260822 27031 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0103 12:45:10.267428 27031 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0103 12:45:10.276458 27031 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/11090.pem && ln -fs /usr/share/ca-certificates/11090.pem /etc/ssl/certs/11090.pem"
I0103 12:45:10.285641 27031 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/11090.pem
I0103 12:45:10.289699 27031 certs.go:480] hashing: -rw-r--r-- 1 root root 1338 Jan 3 19:57 /usr/share/ca-certificates/11090.pem
I0103 12:45:10.289744 27031 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/11090.pem
I0103 12:45:10.296558 27031 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/11090.pem /etc/ssl/certs/51391683.0"
I0103 12:45:10.305710 27031 ssh_runner.go:195] Run: ls /var/lib/minikube/certs/etcd
I0103 12:45:10.310010 27031 certs.go:353] certs directory doesn't exist, likely first start: ls /var/lib/minikube/certs/etcd: Process exited with status 2
stdout:
stderr:
ls: cannot access '/var/lib/minikube/certs/etcd': No such file or directory
I0103 12:45:10.310058 27031 kubeadm.go:404] StartCluster: {Name:old-k8s-version-079000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703498848-17857@sha256:81ae12a49915e4f02aa382dd3758a30a6649e1143c32b3d03309750104577c6c Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-079000 Namespace:default APIServerName:minikubeCA APISer
verNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false Disab
leMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 AutoPauseInterval:1m0s GPUs:}
I0103 12:45:10.310151 27031 ssh_runner.go:195] Run: docker ps --filter status=paused --filter=name=k8s_.*_(kube-system)_ --format={{.ID}}
I0103 12:45:10.327975 27031 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0103 12:45:10.336645 27031 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0103 12:45:10.345068 27031 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0103 12:45:10.345121 27031 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0103 12:45:10.353372 27031 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0103 12:45:10.353405 27031 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0103 12:45:10.403692 27031 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0103 12:45:10.403764 27031 kubeadm.go:322] [preflight] Running pre-flight checks
I0103 12:45:10.658112 27031 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0103 12:45:10.658205 27031 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0103 12:45:10.658283 27031 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0103 12:45:10.866918 27031 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0103 12:45:10.867640 27031 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0103 12:45:10.873795 27031 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0103 12:45:10.942283 27031 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0103 12:45:10.985386 27031 out.go:204] - Generating certificates and keys ...
I0103 12:45:10.985519 27031 kubeadm.go:322] [certs] Using existing ca certificate authority
I0103 12:45:10.985651 27031 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0103 12:45:11.243563 27031 kubeadm.go:322] [certs] Generating "apiserver-kubelet-client" certificate and key
I0103 12:45:11.360551 27031 kubeadm.go:322] [certs] Generating "front-proxy-ca" certificate and key
I0103 12:45:11.414108 27031 kubeadm.go:322] [certs] Generating "front-proxy-client" certificate and key
I0103 12:45:11.537057 27031 kubeadm.go:322] [certs] Generating "etcd/ca" certificate and key
I0103 12:45:11.636525 27031 kubeadm.go:322] [certs] Generating "etcd/server" certificate and key
I0103 12:45:11.636652 27031 kubeadm.go:322] [certs] etcd/server serving cert is signed for DNS names [old-k8s-version-079000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
I0103 12:45:11.759019 27031 kubeadm.go:322] [certs] Generating "etcd/peer" certificate and key
I0103 12:45:11.759246 27031 kubeadm.go:322] [certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-079000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
I0103 12:45:11.886502 27031 kubeadm.go:322] [certs] Generating "etcd/healthcheck-client" certificate and key
I0103 12:45:12.068550 27031 kubeadm.go:322] [certs] Generating "apiserver-etcd-client" certificate and key
I0103 12:45:12.241042 27031 kubeadm.go:322] [certs] Generating "sa" key and public key
I0103 12:45:12.241109 27031 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0103 12:45:12.314947 27031 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0103 12:45:12.586345 27031 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0103 12:45:12.702648 27031 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0103 12:45:12.818204 27031 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0103 12:45:12.818834 27031 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0103 12:45:12.842372 27031 out.go:204] - Booting up control plane ...
I0103 12:45:12.842550 27031 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0103 12:45:12.842662 27031 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0103 12:45:12.842767 27031 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0103 12:45:12.842907 27031 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0103 12:45:12.843186 27031 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0103 12:45:52.827039 27031 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0103 12:45:52.827706 27031 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0103 12:45:52.827870 27031 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0103 12:45:57.828312 27031 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0103 12:45:57.828466 27031 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0103 12:46:07.829236 27031 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0103 12:46:07.829404 27031 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0103 12:46:27.829759 27031 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0103 12:46:27.829958 27031 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0103 12:47:07.830421 27031 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0103 12:47:07.830768 27031 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0103 12:47:07.830933 27031 kubeadm.go:322]
I0103 12:47:07.830984 27031 kubeadm.go:322] Unfortunately, an error has occurred:
I0103 12:47:07.831030 27031 kubeadm.go:322] timed out waiting for the condition
I0103 12:47:07.831036 27031 kubeadm.go:322]
I0103 12:47:07.831076 27031 kubeadm.go:322] This error is likely caused by:
I0103 12:47:07.831122 27031 kubeadm.go:322] - The kubelet is not running
I0103 12:47:07.831241 27031 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0103 12:47:07.831251 27031 kubeadm.go:322]
I0103 12:47:07.831345 27031 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0103 12:47:07.831386 27031 kubeadm.go:322] - 'systemctl status kubelet'
I0103 12:47:07.831427 27031 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0103 12:47:07.831435 27031 kubeadm.go:322]
I0103 12:47:07.831547 27031 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0103 12:47:07.831617 27031 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0103 12:47:07.831709 27031 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0103 12:47:07.831750 27031 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0103 12:47:07.831822 27031 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0103 12:47:07.831856 27031 kubeadm.go:322] - 'docker logs CONTAINERID'
I0103 12:47:07.833122 27031 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0103 12:47:07.833194 27031 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0103 12:47:07.833335 27031 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
I0103 12:47:07.833485 27031 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0103 12:47:07.833626 27031 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0103 12:47:07.833708 27031 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
W0103 12:47:07.833803 27031 out.go:239] ! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-079000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-079000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-079000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-079000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
I0103 12:47:07.833832 27031 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm reset --cri-socket /var/run/dockershim.sock --force"
I0103 12:47:08.245098 27031 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service kubelet
I0103 12:47:08.256129 27031 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0103 12:47:08.256182 27031 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0103 12:47:08.265049 27031 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0103 12:47:08.265070 27031 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0103 12:47:08.316936 27031 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0103 12:47:08.316983 27031 kubeadm.go:322] [preflight] Running pre-flight checks
I0103 12:47:08.552212 27031 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0103 12:47:08.552304 27031 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0103 12:47:08.552424 27031 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0103 12:47:08.726738 27031 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0103 12:47:08.727517 27031 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0103 12:47:08.733454 27031 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0103 12:47:08.801136 27031 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0103 12:47:08.821230 27031 out.go:204] - Generating certificates and keys ...
I0103 12:47:08.821293 27031 kubeadm.go:322] [certs] Using existing ca certificate authority
I0103 12:47:08.821366 27031 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0103 12:47:08.821436 27031 kubeadm.go:322] [certs] Using existing apiserver-kubelet-client certificate and key on disk
I0103 12:47:08.821489 27031 kubeadm.go:322] [certs] Using existing front-proxy-ca certificate authority
I0103 12:47:08.821557 27031 kubeadm.go:322] [certs] Using existing front-proxy-client certificate and key on disk
I0103 12:47:08.821605 27031 kubeadm.go:322] [certs] Using existing etcd/ca certificate authority
I0103 12:47:08.821657 27031 kubeadm.go:322] [certs] Using existing etcd/server certificate and key on disk
I0103 12:47:08.821698 27031 kubeadm.go:322] [certs] Using existing etcd/peer certificate and key on disk
I0103 12:47:08.821771 27031 kubeadm.go:322] [certs] Using existing etcd/healthcheck-client certificate and key on disk
I0103 12:47:08.821837 27031 kubeadm.go:322] [certs] Using existing apiserver-etcd-client certificate and key on disk
I0103 12:47:08.821874 27031 kubeadm.go:322] [certs] Using the existing "sa" key
I0103 12:47:08.821921 27031 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0103 12:47:08.957021 27031 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0103 12:47:09.181613 27031 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0103 12:47:09.339438 27031 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0103 12:47:09.456909 27031 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0103 12:47:09.457324 27031 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0103 12:47:09.478896 27031 out.go:204] - Booting up control plane ...
I0103 12:47:09.479036 27031 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0103 12:47:09.479158 27031 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0103 12:47:09.479289 27031 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0103 12:47:09.479490 27031 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0103 12:47:09.479856 27031 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0103 12:47:49.465944 27031 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0103 12:47:49.467075 27031 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0103 12:47:49.467264 27031 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0103 12:47:54.468194 27031 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0103 12:47:54.468411 27031 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0103 12:48:04.469561 27031 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0103 12:48:04.469794 27031 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0103 12:48:24.471432 27031 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0103 12:48:24.471647 27031 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0103 12:49:04.676908 27031 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0103 12:49:04.677114 27031 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0103 12:49:04.677128 27031 kubeadm.go:322]
I0103 12:49:04.677180 27031 kubeadm.go:322] Unfortunately, an error has occurred:
I0103 12:49:04.677226 27031 kubeadm.go:322] timed out waiting for the condition
I0103 12:49:04.677234 27031 kubeadm.go:322]
I0103 12:49:04.677269 27031 kubeadm.go:322] This error is likely caused by:
I0103 12:49:04.677319 27031 kubeadm.go:322] - The kubelet is not running
I0103 12:49:04.677471 27031 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0103 12:49:04.677493 27031 kubeadm.go:322]
I0103 12:49:04.677626 27031 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0103 12:49:04.677666 27031 kubeadm.go:322] - 'systemctl status kubelet'
I0103 12:49:04.677707 27031 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0103 12:49:04.677718 27031 kubeadm.go:322]
I0103 12:49:04.677839 27031 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0103 12:49:04.677952 27031 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0103 12:49:04.678013 27031 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0103 12:49:04.678049 27031 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0103 12:49:04.678104 27031 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0103 12:49:04.678135 27031 kubeadm.go:322] - 'docker logs CONTAINERID'
I0103 12:49:04.679506 27031 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0103 12:49:04.679580 27031 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0103 12:49:04.679693 27031 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
I0103 12:49:04.679775 27031 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0103 12:49:04.679858 27031 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0103 12:49:04.679917 27031 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
I0103 12:49:04.679964 27031 kubeadm.go:406] StartCluster complete in 3m54.168506428s
I0103 12:49:04.680050 27031 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-apiserver --format={{.ID}}
I0103 12:49:04.697783 27031 logs.go:284] 0 containers: []
W0103 12:49:04.697798 27031 logs.go:286] No container was found matching "kube-apiserver"
I0103 12:49:04.697871 27031 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_etcd --format={{.ID}}
I0103 12:49:04.716774 27031 logs.go:284] 0 containers: []
W0103 12:49:04.716786 27031 logs.go:286] No container was found matching "etcd"
I0103 12:49:04.716854 27031 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_coredns --format={{.ID}}
I0103 12:49:04.737044 27031 logs.go:284] 0 containers: []
W0103 12:49:04.737064 27031 logs.go:286] No container was found matching "coredns"
I0103 12:49:04.737151 27031 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-scheduler --format={{.ID}}
I0103 12:49:04.763048 27031 logs.go:284] 0 containers: []
W0103 12:49:04.763066 27031 logs.go:286] No container was found matching "kube-scheduler"
I0103 12:49:04.763147 27031 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-proxy --format={{.ID}}
I0103 12:49:04.808026 27031 logs.go:284] 0 containers: []
W0103 12:49:04.808041 27031 logs.go:286] No container was found matching "kube-proxy"
I0103 12:49:04.808110 27031 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-controller-manager --format={{.ID}}
I0103 12:49:04.827220 27031 logs.go:284] 0 containers: []
W0103 12:49:04.827235 27031 logs.go:286] No container was found matching "kube-controller-manager"
I0103 12:49:04.827331 27031 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kindnet --format={{.ID}}
I0103 12:49:04.850949 27031 logs.go:284] 0 containers: []
W0103 12:49:04.850965 27031 logs.go:286] No container was found matching "kindnet"
I0103 12:49:04.850980 27031 logs.go:123] Gathering logs for container status ...
I0103 12:49:04.850988 27031 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0103 12:49:04.909376 27031 logs.go:123] Gathering logs for kubelet ...
I0103 12:49:04.909397 27031 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
I0103 12:49:04.946103 27031 logs.go:123] Gathering logs for dmesg ...
I0103 12:49:04.946118 27031 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0103 12:49:04.959573 27031 logs.go:123] Gathering logs for describe nodes ...
I0103 12:49:04.959589 27031 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0103 12:49:05.032189 27031 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0103 12:49:05.032202 27031 logs.go:123] Gathering logs for Docker ...
I0103 12:49:05.032210 27031 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u docker -u cri-docker -n 400"
W0103 12:49:05.107826 27031 out.go:369] Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0103 12:49:05.107854 27031 out.go:239] *
*
W0103 12:49:05.107889 27031 out.go:239] X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0103 12:49:05.107903 27031 out.go:239] *
*
W0103 12:49:05.108509 27031 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0103 12:49:05.170910 27031 out.go:177]
W0103 12:49:05.229089 27031 out.go:239] X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0103 12:49:05.229173 27031 out.go:239] * Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
* Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
W0103 12:49:05.229206 27031 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/4172
* Related issue: https://github.com/kubernetes/minikube/issues/4172
I0103 12:49:05.251134 27031 out.go:177]
** /stderr **
start_stop_delete_test.go:188: failed starting minikube -first start-. args "out/minikube-darwin-amd64 start -p old-k8s-version-079000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0": exit status 109
helpers_test.go:222: -----------------------post-mortem--------------------------------
helpers_test.go:230: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/FirstStart]: docker inspect <======
helpers_test.go:231: (dbg) Run: docker inspect old-k8s-version-079000
helpers_test.go:235: (dbg) docker inspect old-k8s-version-079000:
-- stdout --
[
{
"Id": "488c5550224f7171ef208cbcc14e72b8e9b660b173df8ac004b157f8873115c3",
"Created": "2024-01-03T20:44:55.833825695Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 305065,
"ExitCode": 0,
"Error": "",
"StartedAt": "2024-01-03T20:44:56.088641196Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:b531f61d9bfacb74e25e3fb20a2533b78fa4bf98ca9061755006074ff8c2c789",
"ResolvConfPath": "/var/lib/docker/containers/488c5550224f7171ef208cbcc14e72b8e9b660b173df8ac004b157f8873115c3/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/488c5550224f7171ef208cbcc14e72b8e9b660b173df8ac004b157f8873115c3/hostname",
"HostsPath": "/var/lib/docker/containers/488c5550224f7171ef208cbcc14e72b8e9b660b173df8ac004b157f8873115c3/hosts",
"LogPath": "/var/lib/docker/containers/488c5550224f7171ef208cbcc14e72b8e9b660b173df8ac004b157f8873115c3/488c5550224f7171ef208cbcc14e72b8e9b660b173df8ac004b157f8873115c3-json.log",
"Name": "/old-k8s-version-079000",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro",
"old-k8s-version-079000:/var"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "old-k8s-version-079000",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 2306867200,
"NanoCpus": 2000000000,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": [],
"BlkioDeviceWriteBps": [],
"BlkioDeviceReadIOps": [],
"BlkioDeviceWriteIOps": [],
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 2306867200,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/41397c3f2359361b0e0bf92adf74fbe1a4b9037b18553c584c4e396602c1392a-init/diff:/var/lib/docker/overlay2/d51c25870073ca49ae45bcaffff5d04b6853b273710b15cd26d3414e5d7cfab6/diff",
"MergedDir": "/var/lib/docker/overlay2/41397c3f2359361b0e0bf92adf74fbe1a4b9037b18553c584c4e396602c1392a/merged",
"UpperDir": "/var/lib/docker/overlay2/41397c3f2359361b0e0bf92adf74fbe1a4b9037b18553c584c4e396602c1392a/diff",
"WorkDir": "/var/lib/docker/overlay2/41397c3f2359361b0e0bf92adf74fbe1a4b9037b18553c584c4e396602c1392a/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "old-k8s-version-079000",
"Source": "/var/lib/docker/volumes/old-k8s-version-079000/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "old-k8s-version-079000",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"22/tcp": {},
"2376/tcp": {},
"32443/tcp": {},
"5000/tcp": {},
"8443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1703498848-17857@sha256:81ae12a49915e4f02aa382dd3758a30a6649e1143c32b3d03309750104577c6c",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "old-k8s-version-079000",
"name.minikube.sigs.k8s.io": "old-k8s-version-079000",
"role.minikube.sigs.k8s.io": ""
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "6357951fe2fce563f27bedcc1cb3c39c60c5eacdb7adf6e2528abd3637183942",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "61413"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "61414"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "61415"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "61416"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "61417"
}
]
},
"SandboxKey": "/var/run/docker/netns/6357951fe2fc",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"old-k8s-version-079000": {
"IPAMConfig": {
"IPv4Address": "192.168.76.2"
},
"Links": null,
"Aliases": [
"488c5550224f",
"old-k8s-version-079000"
],
"MacAddress": "02:42:c0:a8:4c:02",
"NetworkID": "fa57a59237dbd216e3611a46ef90c42978dc8b8c11f6ffc7c61970c426e7ce95",
"EndpointID": "3167c2edbedae44e0602e2352e8bbfcda02969d3b3da3871fc35ad6f31fb9b2f",
"Gateway": "192.168.76.1",
"IPAddress": "192.168.76.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"DriverOpts": null
}
}
}
}
]
-- /stdout --
helpers_test.go:239: (dbg) Run: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-079000 -n old-k8s-version-079000
helpers_test.go:239: (dbg) Non-zero exit: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-079000 -n old-k8s-version-079000: exit status 6 (425.571313ms)
-- stdout --
Running
WARNING: Your kubectl is pointing to stale minikube-vm.
To fix the kubectl context, run `minikube update-context`
-- /stdout --
** stderr **
E0103 12:49:05.916799 27932 status.go:415] kubeconfig endpoint: extract IP: "old-k8s-version-079000" does not appear in /Users/jenkins/minikube-integration/17885-10646/kubeconfig
** /stderr **
helpers_test.go:239: status error: exit status 6 (may be ok)
helpers_test.go:241: "old-k8s-version-079000" host is not running, skipping log retrieval (state="Running\nWARNING: Your kubectl is pointing to stale minikube-vm.\nTo fix the kubectl context, run `minikube update-context`")
--- FAIL: TestStartStop/group/old-k8s-version/serial/FirstStart (254.57s)