=== RUN TestStartStop/group/old-k8s-version/serial/FirstStart
start_stop_delete_test.go:186: (dbg) Run: out/minikube-darwin-amd64 start -p old-k8s-version-901000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0
E0108 19:22:23.418902 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/auto-798000/client.crt: no such file or directory
E0108 19:22:25.706048 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/kindnet-798000/client.crt: no such file or directory
E0108 19:22:25.711655 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/kindnet-798000/client.crt: no such file or directory
E0108 19:22:25.721852 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/kindnet-798000/client.crt: no such file or directory
E0108 19:22:25.742652 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/kindnet-798000/client.crt: no such file or directory
E0108 19:22:25.782779 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/kindnet-798000/client.crt: no such file or directory
E0108 19:22:25.862912 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/kindnet-798000/client.crt: no such file or directory
E0108 19:22:26.023331 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/kindnet-798000/client.crt: no such file or directory
E0108 19:22:26.343652 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/kindnet-798000/client.crt: no such file or directory
E0108 19:22:26.983844 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/kindnet-798000/client.crt: no such file or directory
E0108 19:22:28.264160 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/kindnet-798000/client.crt: no such file or directory
E0108 19:22:30.824240 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/kindnet-798000/client.crt: no such file or directory
E0108 19:22:35.944243 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/kindnet-798000/client.crt: no such file or directory
E0108 19:22:43.899545 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/auto-798000/client.crt: no such file or directory
E0108 19:22:46.185764 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/kindnet-798000/client.crt: no such file or directory
E0108 19:23:06.665464 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/kindnet-798000/client.crt: no such file or directory
start_stop_delete_test.go:186: (dbg) Non-zero exit: out/minikube-darwin-amd64 start -p old-k8s-version-901000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0: exit status 109 (4m13.080370557s)
-- stdout --
* [old-k8s-version-901000] minikube v1.32.0 on Darwin 14.2.1
- MINIKUBE_LOCATION=17866
- KUBECONFIG=/Users/jenkins/minikube-integration/17866-74927/kubeconfig
- MINIKUBE_BIN=out/minikube-darwin-amd64
- MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
- MINIKUBE_HOME=/Users/jenkins/minikube-integration/17866-74927/.minikube
- MINIKUBE_FORCE_SYSTEMD=
* Using the docker driver based on user configuration
* Using Docker Desktop driver with root privileges
* Starting control plane node old-k8s-version-901000 in cluster old-k8s-version-901000
* Pulling base image v0.0.42-1704759386-17866 ...
* Creating docker container (CPUs=2, Memory=2200MB) ...
* Preparing Kubernetes v1.16.0 on Docker 24.0.7 ...
- Generating certificates and keys ...
- Booting up control plane ...
- Generating certificates and keys ...
- Booting up control plane ...
-- /stdout --
** stderr **
I0108 19:22:23.388239 90261 out.go:296] Setting OutFile to fd 1 ...
I0108 19:22:23.388571 90261 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0108 19:22:23.388583 90261 out.go:309] Setting ErrFile to fd 2...
I0108 19:22:23.388588 90261 out.go:343] TERM=,COLORTERM=, which probably does not support color
I0108 19:22:23.388814 90261 root.go:338] Updating PATH: /Users/jenkins/minikube-integration/17866-74927/.minikube/bin
I0108 19:22:23.390417 90261 out.go:303] Setting JSON to false
I0108 19:22:23.414241 90261 start.go:128] hostinfo: {"hostname":"MacOS-Agent-2.local","uptime":37315,"bootTime":1704733228,"procs":482,"os":"darwin","platform":"darwin","platformFamily":"Standalone Workstation","platformVersion":"14.2.1","kernelVersion":"23.2.0","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"2965c349-98a5-5970-aaa9-9eedd3ae5959"}
W0108 19:22:23.414348 90261 start.go:136] gopshost.Virtualization returned error: not implemented yet
I0108 19:22:23.435653 90261 out.go:177] * [old-k8s-version-901000] minikube v1.32.0 on Darwin 14.2.1
I0108 19:22:23.477212 90261 out.go:177] - MINIKUBE_LOCATION=17866
I0108 19:22:23.498445 90261 out.go:177] - KUBECONFIG=/Users/jenkins/minikube-integration/17866-74927/kubeconfig
I0108 19:22:23.477305 90261 notify.go:220] Checking for updates...
I0108 19:22:23.540170 90261 out.go:177] - MINIKUBE_BIN=out/minikube-darwin-amd64
I0108 19:22:23.561291 90261 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0108 19:22:23.582436 90261 out.go:177] - MINIKUBE_HOME=/Users/jenkins/minikube-integration/17866-74927/.minikube
I0108 19:22:23.603228 90261 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0108 19:22:23.624945 90261 config.go:182] Loaded profile config "kubenet-798000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.28.4
I0108 19:22:23.625046 90261 driver.go:392] Setting default libvirt URI to qemu:///system
I0108 19:22:23.683789 90261 docker.go:122] docker version: linux-24.0.7:Docker Desktop 4.26.0 (130397)
I0108 19:22:23.683953 90261 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0108 19:22:23.793522 90261 info.go:266] docker info: {ID:9dd12a49-41d2-44e8-aa64-4ab7fa99394e Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:58 OomKillDisable:false NGoroutines:68 SystemTime:2024-01-09 03:22:23.782697356 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:8 KernelVersion:6.5.11-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServe
rAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:12 MemTotal:6221275136 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:24.0.7 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f Expected:d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f} RuncCommit:{ID:v1.1.10-0-g18a0cb0 Expected:v1.1.10-0-g18a0cb0} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=unconf
ined name=cgroupns] ProductLicense: Warnings:[WARNING: daemon is not using the default seccomp profile] ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.0-desktop.2] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.23.3-desktop.2] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker Dev Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Mana
ges Docker extensions Vendor:Docker Inc. Version:v0.2.21] map[Name:feedback Path:/Users/jenkins/.docker/cli-plugins/docker-feedback SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-feedback] ShortDescription:Provide feedback, right in your terminal! Vendor:Docker Inc. Version:0.1] map[Name:init Path:/Users/jenkins/.docker/cli-plugins/docker-init SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-init] ShortDescription:Creates Docker-related starter files for your project Vendor:Docker Inc. Version:v0.1.0-beta.10] map[Name:sbom Path:/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM) for an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/d
ocker-scan] ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.26.0] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Docker Scout Vendor:Docker Inc. Version:v1.2.0]] Warnings:<nil>}}
I0108 19:22:23.835736 90261 out.go:177] * Using the docker driver based on user configuration
I0108 19:22:23.856767 90261 start.go:298] selected driver: docker
I0108 19:22:23.856780 90261 start.go:902] validating driver "docker" against <nil>
I0108 19:22:23.856788 90261 start.go:913] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0108 19:22:23.860191 90261 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0108 19:22:23.969312 90261 info.go:266] docker info: {ID:9dd12a49-41d2-44e8-aa64-4ab7fa99394e Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:58 OomKillDisable:false NGoroutines:68 SystemTime:2024-01-09 03:22:23.959744735 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:8 KernelVersion:6.5.11-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServe
rAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:12 MemTotal:6221275136 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:24.0.7 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f Expected:d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f} RuncCommit:{ID:v1.1.10-0-g18a0cb0 Expected:v1.1.10-0-g18a0cb0} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=unconf
ined name=cgroupns] ProductLicense: Warnings:[WARNING: daemon is not using the default seccomp profile] ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.0-desktop.2] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.23.3-desktop.2] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker Dev Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Mana
ges Docker extensions Vendor:Docker Inc. Version:v0.2.21] map[Name:feedback Path:/Users/jenkins/.docker/cli-plugins/docker-feedback SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-feedback] ShortDescription:Provide feedback, right in your terminal! Vendor:Docker Inc. Version:0.1] map[Name:init Path:/Users/jenkins/.docker/cli-plugins/docker-init SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-init] ShortDescription:Creates Docker-related starter files for your project Vendor:Docker Inc. Version:v0.1.0-beta.10] map[Name:sbom Path:/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM) for an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/d
ocker-scan] ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.26.0] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Docker Scout Vendor:Docker Inc. Version:v1.2.0]] Warnings:<nil>}}
I0108 19:22:23.969510 90261 start_flags.go:307] no existing cluster config was found, will generate one from the flags
I0108 19:22:23.969694 90261 start_flags.go:927] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0108 19:22:23.991201 90261 out.go:177] * Using Docker Desktop driver with root privileges
I0108 19:22:24.011933 90261 cni.go:84] Creating CNI manager for ""
I0108 19:22:24.011951 90261 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0108 19:22:24.011962 90261 start_flags.go:321] config:
{Name:old-k8s-version-901000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-901000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local Containe
rRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs:}
I0108 19:22:24.032955 90261 out.go:177] * Starting control plane node old-k8s-version-901000 in cluster old-k8s-version-901000
I0108 19:22:24.074754 90261 cache.go:121] Beginning downloading kic base image for docker with docker
I0108 19:22:24.095925 90261 out.go:177] * Pulling base image v0.0.42-1704759386-17866 ...
I0108 19:22:24.116991 90261 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0108 19:22:24.117014 90261 image.go:79] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 in local docker daemon
I0108 19:22:24.117032 90261 preload.go:148] Found local preload: /Users/jenkins/minikube-integration/17866-74927/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4
I0108 19:22:24.117045 90261 cache.go:56] Caching tarball of preloaded images
I0108 19:22:24.117168 90261 preload.go:174] Found /Users/jenkins/minikube-integration/17866-74927/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0108 19:22:24.117178 90261 cache.go:59] Finished verifying existence of preloaded tar for v1.16.0 on docker
I0108 19:22:24.117606 90261 profile.go:148] Saving config to /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/config.json ...
I0108 19:22:24.117759 90261 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/config.json: {Name:mk061b505f4d8e86d78dcdcffd8710c353990da4 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0108 19:22:24.169504 90261 image.go:83] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 in local docker daemon, skipping pull
I0108 19:22:24.169522 90261 cache.go:144] gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 exists in daemon, skipping load
I0108 19:22:24.169544 90261 cache.go:194] Successfully downloaded all kic artifacts
I0108 19:22:24.169588 90261 start.go:365] acquiring machines lock for old-k8s-version-901000: {Name:mk41257d6f9820536f749153d111ded94c6d377e Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0108 19:22:24.169770 90261 start.go:369] acquired machines lock for "old-k8s-version-901000" in 168.915µs
I0108 19:22:24.169798 90261 start.go:93] Provisioning new machine with config: &{Name:old-k8s-version-901000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-901000 Namespace:default APIServer
Name:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:f
alse DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs:} &{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}
I0108 19:22:24.169954 90261 start.go:125] createHost starting for "" (driver="docker")
I0108 19:22:24.214407 90261 out.go:204] * Creating docker container (CPUs=2, Memory=2200MB) ...
I0108 19:22:24.214616 90261 start.go:159] libmachine.API.Create for "old-k8s-version-901000" (driver="docker")
I0108 19:22:24.214644 90261 client.go:168] LocalClient.Create starting
I0108 19:22:24.214747 90261 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/17866-74927/.minikube/certs/ca.pem
I0108 19:22:24.214794 90261 main.go:141] libmachine: Decoding PEM data...
I0108 19:22:24.214813 90261 main.go:141] libmachine: Parsing certificate...
I0108 19:22:24.214870 90261 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/17866-74927/.minikube/certs/cert.pem
I0108 19:22:24.214905 90261 main.go:141] libmachine: Decoding PEM data...
I0108 19:22:24.214913 90261 main.go:141] libmachine: Parsing certificate...
I0108 19:22:24.215381 90261 cli_runner.go:164] Run: docker network inspect old-k8s-version-901000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
W0108 19:22:24.267770 90261 cli_runner.go:211] docker network inspect old-k8s-version-901000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1
I0108 19:22:24.267866 90261 network_create.go:281] running [docker network inspect old-k8s-version-901000] to gather additional debugging logs...
I0108 19:22:24.267885 90261 cli_runner.go:164] Run: docker network inspect old-k8s-version-901000
W0108 19:22:24.320255 90261 cli_runner.go:211] docker network inspect old-k8s-version-901000 returned with exit code 1
I0108 19:22:24.320283 90261 network_create.go:284] error running [docker network inspect old-k8s-version-901000]: docker network inspect old-k8s-version-901000: exit status 1
stdout:
[]
stderr:
Error response from daemon: network old-k8s-version-901000 not found
I0108 19:22:24.320303 90261 network_create.go:286] output of [docker network inspect old-k8s-version-901000]: -- stdout --
[]
-- /stdout --
** stderr **
Error response from daemon: network old-k8s-version-901000 not found
** /stderr **
I0108 19:22:24.320448 90261 cli_runner.go:164] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0108 19:22:24.374576 90261 network.go:212] skipping subnet 192.168.49.0/24 that is reserved: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0108 19:22:24.376146 90261 network.go:212] skipping subnet 192.168.58.0/24 that is reserved: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0108 19:22:24.377717 90261 network.go:212] skipping subnet 192.168.67.0/24 that is reserved: &{IP:192.168.67.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.67.0/24 Gateway:192.168.67.1 ClientMin:192.168.67.2 ClientMax:192.168.67.254 Broadcast:192.168.67.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0108 19:22:24.379088 90261 network.go:212] skipping subnet 192.168.76.0/24 that is reserved: &{IP:192.168.76.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.76.0/24 Gateway:192.168.76.1 ClientMin:192.168.76.2 ClientMax:192.168.76.254 Broadcast:192.168.76.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0108 19:22:24.379469 90261 network.go:209] using free private subnet 192.168.85.0/24: &{IP:192.168.85.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.85.0/24 Gateway:192.168.85.1 ClientMin:192.168.85.2 ClientMax:192.168.85.254 Broadcast:192.168.85.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc002404c90}
I0108 19:22:24.379485 90261 network_create.go:124] attempt to create docker network old-k8s-version-901000 192.168.85.0/24 with gateway 192.168.85.1 and MTU of 65535 ...
I0108 19:22:24.379557 90261 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.85.0/24 --gateway=192.168.85.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-901000 old-k8s-version-901000
I0108 19:22:24.468838 90261 network_create.go:108] docker network old-k8s-version-901000 192.168.85.0/24 created
I0108 19:22:24.468904 90261 kic.go:121] calculated static IP "192.168.85.2" for the "old-k8s-version-901000" container
I0108 19:22:24.469007 90261 cli_runner.go:164] Run: docker ps -a --format {{.Names}}
I0108 19:22:24.521373 90261 cli_runner.go:164] Run: docker volume create old-k8s-version-901000 --label name.minikube.sigs.k8s.io=old-k8s-version-901000 --label created_by.minikube.sigs.k8s.io=true
I0108 19:22:24.574175 90261 oci.go:103] Successfully created a docker volume old-k8s-version-901000
I0108 19:22:24.574292 90261 cli_runner.go:164] Run: docker run --rm --name old-k8s-version-901000-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-901000 --entrypoint /usr/bin/test -v old-k8s-version-901000:/var gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 -d /var/lib
I0108 19:22:25.105998 90261 oci.go:107] Successfully prepared a docker volume old-k8s-version-901000
I0108 19:22:25.106044 90261 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0108 19:22:25.106059 90261 kic.go:194] Starting extracting preloaded images to volume ...
I0108 19:22:25.106178 90261 cli_runner.go:164] Run: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/17866-74927/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-901000:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 -I lz4 -xf /preloaded.tar -C /extractDir
I0108 19:22:27.493466 90261 cli_runner.go:217] Completed: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/17866-74927/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-901000:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 -I lz4 -xf /preloaded.tar -C /extractDir: (2.3872886s)
I0108 19:22:27.493495 90261 kic.go:203] duration metric: took 2.387489 seconds to extract preloaded images to volume
I0108 19:22:27.493597 90261 cli_runner.go:164] Run: docker info --format "'{{json .SecurityOptions}}'"
I0108 19:22:27.619159 90261 cli_runner.go:164] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname old-k8s-version-901000 --name old-k8s-version-901000 --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-901000 --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=old-k8s-version-901000 --network old-k8s-version-901000 --ip 192.168.85.2 --volume old-k8s-version-901000:/var --security-opt apparmor=unconfined --memory=2200mb --memory-swap=2200mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0
I0108 19:22:27.958607 90261 cli_runner.go:164] Run: docker container inspect old-k8s-version-901000 --format={{.State.Running}}
I0108 19:22:28.025802 90261 cli_runner.go:164] Run: docker container inspect old-k8s-version-901000 --format={{.State.Status}}
I0108 19:22:28.087434 90261 cli_runner.go:164] Run: docker exec old-k8s-version-901000 stat /var/lib/dpkg/alternatives/iptables
I0108 19:22:28.264991 90261 oci.go:144] the created container "old-k8s-version-901000" has a running status.
I0108 19:22:28.265023 90261 kic.go:225] Creating ssh key for kic: /Users/jenkins/minikube-integration/17866-74927/.minikube/machines/old-k8s-version-901000/id_rsa...
I0108 19:22:28.656732 90261 kic_runner.go:191] docker (temp): /Users/jenkins/minikube-integration/17866-74927/.minikube/machines/old-k8s-version-901000/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0108 19:22:28.718865 90261 cli_runner.go:164] Run: docker container inspect old-k8s-version-901000 --format={{.State.Status}}
I0108 19:22:28.776936 90261 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0108 19:22:28.776957 90261 kic_runner.go:114] Args: [docker exec --privileged old-k8s-version-901000 chown docker:docker /home/docker/.ssh/authorized_keys]
I0108 19:22:28.877048 90261 cli_runner.go:164] Run: docker container inspect old-k8s-version-901000 --format={{.State.Status}}
I0108 19:22:28.930425 90261 machine.go:88] provisioning docker machine ...
I0108 19:22:28.930483 90261 ubuntu.go:169] provisioning hostname "old-k8s-version-901000"
I0108 19:22:28.930586 90261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-901000
I0108 19:22:28.981587 90261 main.go:141] libmachine: Using SSH client type: native
I0108 19:22:28.981915 90261 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x14074c0] 0x140a1a0 <nil> [] 0s} 127.0.0.1 49929 <nil> <nil>}
I0108 19:22:28.981927 90261 main.go:141] libmachine: About to run SSH command:
sudo hostname old-k8s-version-901000 && echo "old-k8s-version-901000" | sudo tee /etc/hostname
I0108 19:22:29.124159 90261 main.go:141] libmachine: SSH cmd err, output: <nil>: old-k8s-version-901000
I0108 19:22:29.124260 90261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-901000
I0108 19:22:29.176984 90261 main.go:141] libmachine: Using SSH client type: native
I0108 19:22:29.177283 90261 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x14074c0] 0x140a1a0 <nil> [] 0s} 127.0.0.1 49929 <nil> <nil>}
I0108 19:22:29.177296 90261 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sold-k8s-version-901000' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 old-k8s-version-901000/g' /etc/hosts;
else
echo '127.0.1.1 old-k8s-version-901000' | sudo tee -a /etc/hosts;
fi
fi
I0108 19:22:29.310684 90261 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0108 19:22:29.310705 90261 ubuntu.go:175] set auth options {CertDir:/Users/jenkins/minikube-integration/17866-74927/.minikube CaCertPath:/Users/jenkins/minikube-integration/17866-74927/.minikube/certs/ca.pem CaPrivateKeyPath:/Users/jenkins/minikube-integration/17866-74927/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/Users/jenkins/minikube-integration/17866-74927/.minikube/machines/server.pem ServerKeyPath:/Users/jenkins/minikube-integration/17866-74927/.minikube/machines/server-key.pem ClientKeyPath:/Users/jenkins/minikube-integration/17866-74927/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/Users/jenkins/minikube-integration/17866-74927/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/Users/jenkins/minikube-integration/17866-74927/.minikube}
I0108 19:22:29.310727 90261 ubuntu.go:177] setting up certificates
I0108 19:22:29.310735 90261 provision.go:83] configureAuth start
I0108 19:22:29.310807 90261 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-901000
I0108 19:22:29.361767 90261 provision.go:138] copyHostCerts
I0108 19:22:29.361869 90261 exec_runner.go:144] found /Users/jenkins/minikube-integration/17866-74927/.minikube/ca.pem, removing ...
I0108 19:22:29.361878 90261 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/17866-74927/.minikube/ca.pem
I0108 19:22:29.361998 90261 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/17866-74927/.minikube/certs/ca.pem --> /Users/jenkins/minikube-integration/17866-74927/.minikube/ca.pem (1078 bytes)
I0108 19:22:29.362215 90261 exec_runner.go:144] found /Users/jenkins/minikube-integration/17866-74927/.minikube/cert.pem, removing ...
I0108 19:22:29.362222 90261 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/17866-74927/.minikube/cert.pem
I0108 19:22:29.362295 90261 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/17866-74927/.minikube/certs/cert.pem --> /Users/jenkins/minikube-integration/17866-74927/.minikube/cert.pem (1123 bytes)
I0108 19:22:29.362458 90261 exec_runner.go:144] found /Users/jenkins/minikube-integration/17866-74927/.minikube/key.pem, removing ...
I0108 19:22:29.362464 90261 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/17866-74927/.minikube/key.pem
I0108 19:22:29.362537 90261 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/17866-74927/.minikube/certs/key.pem --> /Users/jenkins/minikube-integration/17866-74927/.minikube/key.pem (1679 bytes)
I0108 19:22:29.362681 90261 provision.go:112] generating server cert: /Users/jenkins/minikube-integration/17866-74927/.minikube/machines/server.pem ca-key=/Users/jenkins/minikube-integration/17866-74927/.minikube/certs/ca.pem private-key=/Users/jenkins/minikube-integration/17866-74927/.minikube/certs/ca-key.pem org=jenkins.old-k8s-version-901000 san=[192.168.85.2 127.0.0.1 localhost 127.0.0.1 minikube old-k8s-version-901000]
I0108 19:22:29.475890 90261 provision.go:172] copyRemoteCerts
I0108 19:22:29.475949 90261 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0108 19:22:29.476002 90261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-901000
I0108 19:22:29.527576 90261 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49929 SSHKeyPath:/Users/jenkins/minikube-integration/17866-74927/.minikube/machines/old-k8s-version-901000/id_rsa Username:docker}
I0108 19:22:29.622168 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1078 bytes)
I0108 19:22:29.645591 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/machines/server.pem --> /etc/docker/server.pem (1241 bytes)
I0108 19:22:29.668531 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1679 bytes)
I0108 19:22:29.690992 90261 provision.go:86] duration metric: configureAuth took 380.252038ms
I0108 19:22:29.691007 90261 ubuntu.go:193] setting minikube options for container-runtime
I0108 19:22:29.691156 90261 config.go:182] Loaded profile config "old-k8s-version-901000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.16.0
I0108 19:22:29.691222 90261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-901000
I0108 19:22:29.751561 90261 main.go:141] libmachine: Using SSH client type: native
I0108 19:22:29.751927 90261 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x14074c0] 0x140a1a0 <nil> [] 0s} 127.0.0.1 49929 <nil> <nil>}
I0108 19:22:29.751944 90261 main.go:141] libmachine: About to run SSH command:
df --output=fstype / | tail -n 1
I0108 19:22:29.888237 90261 main.go:141] libmachine: SSH cmd err, output: <nil>: overlay
I0108 19:22:29.888253 90261 ubuntu.go:71] root file system type: overlay
I0108 19:22:29.888386 90261 provision.go:309] Updating docker unit: /lib/systemd/system/docker.service ...
I0108 19:22:29.888486 90261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-901000
I0108 19:22:29.951221 90261 main.go:141] libmachine: Using SSH client type: native
I0108 19:22:29.951616 90261 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x14074c0] 0x140a1a0 <nil> [] 0s} 127.0.0.1 49929 <nil> <nil>}
I0108 19:22:29.951672 90261 main.go:141] libmachine: About to run SSH command:
sudo mkdir -p /lib/systemd/system && printf %s "[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP \$MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service.new
I0108 19:22:30.096754 90261 main.go:141] libmachine: SSH cmd err, output: <nil>: [Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
I0108 19:22:30.096856 90261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-901000
I0108 19:22:30.149094 90261 main.go:141] libmachine: Using SSH client type: native
I0108 19:22:30.149385 90261 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x14074c0] 0x140a1a0 <nil> [] 0s} 127.0.0.1 49929 <nil> <nil>}
I0108 19:22:30.149398 90261 main.go:141] libmachine: About to run SSH command:
sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
I0108 19:22:30.718594 90261 main.go:141] libmachine: SSH cmd err, output: <nil>: --- /lib/systemd/system/docker.service 2023-10-26 09:06:22.000000000 +0000
+++ /lib/systemd/system/docker.service.new 2024-01-09 03:22:30.094959835 +0000
@@ -1,30 +1,32 @@
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
-After=network-online.target docker.socket firewalld.service containerd.service time-set.target
-Wants=network-online.target containerd.service
+BindsTo=containerd.service
+After=network-online.target firewalld.service containerd.service
+Wants=network-online.target
Requires=docker.socket
+StartLimitBurst=3
+StartLimitIntervalSec=60
[Service]
Type=notify
-# the default is not to use systemd for cgroups because the delegate issues still
-# exists and systemd currently does not support the cgroup feature set required
-# for containers run by docker
-ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
-ExecReload=/bin/kill -s HUP $MAINPID
-TimeoutStartSec=0
-RestartSec=2
-Restart=always
+Restart=on-failure
-# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
-# Both the old, and new location are accepted by systemd 229 and up, so using the old location
-# to make them work for either version of systemd.
-StartLimitBurst=3
-# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
-# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
-# this option work for either version of systemd.
-StartLimitInterval=60s
+
+# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
+# The base configuration already specifies an 'ExecStart=...' command. The first directive
+# here is to clear out that command inherited from the base configuration. Without this,
+# the command from the base configuration and the command specified here are treated as
+# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
+# will catch this invalid input and refuse to start the service with an error like:
+# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
+
+# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
+# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
+ExecStart=
+ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
+ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
@@ -32,16 +34,16 @@
LimitNPROC=infinity
LimitCORE=infinity
-# Comment TasksMax if your systemd version does not support it.
-# Only systemd 226 and above support this option.
+# Uncomment TasksMax if your systemd version supports it.
+# Only systemd 226 and above support this version.
TasksMax=infinity
+TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
-OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
I0108 19:22:30.718629 90261 machine.go:91] provisioned docker machine in 1.788211026s
I0108 19:22:30.718638 90261 client.go:171] LocalClient.Create took 6.504130733s
I0108 19:22:30.718680 90261 start.go:167] duration metric: libmachine.API.Create for "old-k8s-version-901000" took 6.504204726s
I0108 19:22:30.718693 90261 start.go:300] post-start starting for "old-k8s-version-901000" (driver="docker")
I0108 19:22:30.718703 90261 start.go:329] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0108 19:22:30.718795 90261 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0108 19:22:30.718888 90261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-901000
I0108 19:22:30.775848 90261 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49929 SSHKeyPath:/Users/jenkins/minikube-integration/17866-74927/.minikube/machines/old-k8s-version-901000/id_rsa Username:docker}
I0108 19:22:30.870010 90261 ssh_runner.go:195] Run: cat /etc/os-release
I0108 19:22:30.874631 90261 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0108 19:22:30.874655 90261 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0108 19:22:30.874663 90261 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0108 19:22:30.874668 90261 info.go:137] Remote host: Ubuntu 22.04.3 LTS
I0108 19:22:30.874684 90261 filesync.go:126] Scanning /Users/jenkins/minikube-integration/17866-74927/.minikube/addons for local assets ...
I0108 19:22:30.874791 90261 filesync.go:126] Scanning /Users/jenkins/minikube-integration/17866-74927/.minikube/files for local assets ...
I0108 19:22:30.874985 90261 filesync.go:149] local asset: /Users/jenkins/minikube-integration/17866-74927/.minikube/files/etc/ssl/certs/753692.pem -> 753692.pem in /etc/ssl/certs
I0108 19:22:30.875196 90261 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0108 19:22:30.883400 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/files/etc/ssl/certs/753692.pem --> /etc/ssl/certs/753692.pem (1708 bytes)
I0108 19:22:30.903840 90261 start.go:303] post-start completed in 185.1418ms
I0108 19:22:30.904418 90261 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-901000
I0108 19:22:30.956155 90261 profile.go:148] Saving config to /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/config.json ...
I0108 19:22:30.956621 90261 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0108 19:22:30.956679 90261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-901000
I0108 19:22:31.007721 90261 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49929 SSHKeyPath:/Users/jenkins/minikube-integration/17866-74927/.minikube/machines/old-k8s-version-901000/id_rsa Username:docker}
I0108 19:22:31.099996 90261 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0108 19:22:31.104741 90261 start.go:128] duration metric: createHost completed in 6.93492105s
I0108 19:22:31.104763 90261 start.go:83] releasing machines lock for "old-k8s-version-901000", held for 6.935137574s
I0108 19:22:31.104856 90261 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-901000
I0108 19:22:31.156547 90261 ssh_runner.go:195] Run: cat /version.json
I0108 19:22:31.156571 90261 ssh_runner.go:195] Run: curl -sS -m 2 https://registry.k8s.io/
I0108 19:22:31.156627 90261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-901000
I0108 19:22:31.156654 90261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-901000
I0108 19:22:31.210419 90261 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49929 SSHKeyPath:/Users/jenkins/minikube-integration/17866-74927/.minikube/machines/old-k8s-version-901000/id_rsa Username:docker}
I0108 19:22:31.210415 90261 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49929 SSHKeyPath:/Users/jenkins/minikube-integration/17866-74927/.minikube/machines/old-k8s-version-901000/id_rsa Username:docker}
I0108 19:22:31.305282 90261 ssh_runner.go:195] Run: systemctl --version
I0108 19:22:31.453114 90261 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
I0108 19:22:31.458272 90261 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *loopback.conf* -not -name *.mk_disabled -exec sh -c "grep -q loopback {} && ( grep -q name {} || sudo sed -i '/"type": "loopback"/i \ \ \ \ "name": "loopback",' {} ) && sudo sed -i 's|"cniVersion": ".*"|"cniVersion": "1.0.0"|g' {}" ;
I0108 19:22:31.484603 90261 cni.go:230] loopback cni configuration patched: "/etc/cni/net.d/*loopback.conf*" found
I0108 19:22:31.484709 90261 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *bridge* -not -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e '/"dst": ".*:.*"/d' -e 's|^(.*)"dst": (.*)[,*]$|\1"dst": \2|g' -e '/"subnet": ".*:.*"/d' -e 's|^(.*)"subnet": ".*"(.*)[,*]$|\1"subnet": "10.244.0.0/16"\2|g' {}" ;
I0108 19:22:31.503497 90261 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e 's|^(.*)"subnet": ".*"(.*)$|\1"subnet": "10.244.0.0/16"\2|g' -e 's|^(.*)"gateway": ".*"(.*)$|\1"gateway": "10.244.0.1"\2|g' {}" ;
I0108 19:22:31.522324 90261 cni.go:308] configured [/etc/cni/net.d/100-crio-bridge.conf, /etc/cni/net.d/87-podman-bridge.conflist] bridge cni config(s)
I0108 19:22:31.522342 90261 start.go:475] detecting cgroup driver to use...
I0108 19:22:31.522355 90261 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0108 19:22:31.522466 90261 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0108 19:22:31.539362 90261 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.1"|' /etc/containerd/config.toml"
I0108 19:22:31.550183 90261 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I0108 19:22:31.560612 90261 containerd.go:145] configuring containerd to use "cgroupfs" as cgroup driver...
I0108 19:22:31.560702 90261 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = false|g' /etc/containerd/config.toml"
I0108 19:22:31.574470 90261 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0108 19:22:31.586122 90261 ssh_runner.go:195] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I0108 19:22:31.595219 90261 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0108 19:22:31.604316 90261 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0108 19:22:31.616280 90261 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I0108 19:22:31.630668 90261 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0108 19:22:31.643924 90261 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0108 19:22:31.654190 90261 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0108 19:22:31.703319 90261 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0108 19:22:31.868950 90261 start.go:475] detecting cgroup driver to use...
I0108 19:22:31.868973 90261 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0108 19:22:31.869042 90261 ssh_runner.go:195] Run: sudo systemctl cat docker.service
I0108 19:22:31.893756 90261 cruntime.go:279] skipping containerd shutdown because we are bound to it
I0108 19:22:31.893823 90261 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0108 19:22:31.905317 90261 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///var/run/dockershim.sock
" | sudo tee /etc/crictl.yaml"
I0108 19:22:31.927805 90261 ssh_runner.go:195] Run: which cri-dockerd
I0108 19:22:31.934265 90261 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/cri-docker.service.d
I0108 19:22:31.946227 90261 ssh_runner.go:362] scp memory --> /etc/systemd/system/cri-docker.service.d/10-cni.conf (189 bytes)
I0108 19:22:31.968200 90261 ssh_runner.go:195] Run: sudo systemctl unmask docker.service
I0108 19:22:32.050920 90261 ssh_runner.go:195] Run: sudo systemctl enable docker.socket
I0108 19:22:32.111809 90261 docker.go:560] configuring docker to use "cgroupfs" as cgroup driver...
I0108 19:22:32.111899 90261 ssh_runner.go:362] scp memory --> /etc/docker/daemon.json (130 bytes)
I0108 19:22:32.129267 90261 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0108 19:22:32.201595 90261 ssh_runner.go:195] Run: sudo systemctl restart docker
I0108 19:22:32.438737 90261 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0108 19:22:32.461724 90261 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0108 19:22:32.508678 90261 out.go:204] * Preparing Kubernetes v1.16.0 on Docker 24.0.7 ...
I0108 19:22:32.508762 90261 cli_runner.go:164] Run: docker exec -t old-k8s-version-901000 dig +short host.docker.internal
I0108 19:22:32.624551 90261 network.go:96] got host ip for mount in container by digging dns: 192.168.65.254
I0108 19:22:32.624651 90261 ssh_runner.go:195] Run: grep 192.168.65.254 host.minikube.internal$ /etc/hosts
I0108 19:22:32.629514 90261 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.65.254 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0108 19:22:32.641958 90261 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "8443/tcp") 0).HostPort}}'" old-k8s-version-901000
I0108 19:22:32.695910 90261 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0108 19:22:32.695991 90261 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0108 19:22:32.714518 90261 docker.go:671] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0108 19:22:32.714550 90261 docker.go:677] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0108 19:22:32.714617 90261 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0108 19:22:32.724567 90261 ssh_runner.go:195] Run: which lz4
I0108 19:22:32.729846 90261 ssh_runner.go:195] Run: stat -c "%s %y" /preloaded.tar.lz4
I0108 19:22:32.734593 90261 ssh_runner.go:352] existence check for /preloaded.tar.lz4: stat -c "%s %y" /preloaded.tar.lz4: Process exited with status 1
stdout:
stderr:
stat: cannot statx '/preloaded.tar.lz4': No such file or directory
I0108 19:22:32.734627 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 --> /preloaded.tar.lz4 (369789069 bytes)
I0108 19:22:37.804168 90261 docker.go:635] Took 5.074499 seconds to copy over tarball
I0108 19:22:37.804246 90261 ssh_runner.go:195] Run: sudo tar --xattrs --xattrs-include security.capability -I lz4 -C /var -xf /preloaded.tar.lz4
I0108 19:22:39.364005 90261 ssh_runner.go:235] Completed: sudo tar --xattrs --xattrs-include security.capability -I lz4 -C /var -xf /preloaded.tar.lz4: (1.559761147s)
I0108 19:22:39.364025 90261 ssh_runner.go:146] rm: /preloaded.tar.lz4
I0108 19:22:39.402360 90261 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0108 19:22:39.410841 90261 ssh_runner.go:362] scp memory --> /var/lib/docker/image/overlay2/repositories.json (2499 bytes)
I0108 19:22:39.426286 90261 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0108 19:22:39.481164 90261 ssh_runner.go:195] Run: sudo systemctl restart docker
I0108 19:22:40.002417 90261 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0108 19:22:40.020780 90261 docker.go:671] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0108 19:22:40.020796 90261 docker.go:677] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0108 19:22:40.020807 90261 cache_images.go:88] LoadImages start: [registry.k8s.io/kube-apiserver:v1.16.0 registry.k8s.io/kube-controller-manager:v1.16.0 registry.k8s.io/kube-scheduler:v1.16.0 registry.k8s.io/kube-proxy:v1.16.0 registry.k8s.io/pause:3.1 registry.k8s.io/etcd:3.3.15-0 registry.k8s.io/coredns:1.6.2 gcr.io/k8s-minikube/storage-provisioner:v5]
I0108 19:22:40.025921 90261 image.go:134] retrieving image: gcr.io/k8s-minikube/storage-provisioner:v5
I0108 19:22:40.026139 90261 image.go:134] retrieving image: registry.k8s.io/kube-apiserver:v1.16.0
I0108 19:22:40.026255 90261 image.go:134] retrieving image: registry.k8s.io/kube-proxy:v1.16.0
I0108 19:22:40.026372 90261 image.go:134] retrieving image: registry.k8s.io/kube-controller-manager:v1.16.0
I0108 19:22:40.026449 90261 image.go:134] retrieving image: registry.k8s.io/etcd:3.3.15-0
I0108 19:22:40.026454 90261 image.go:134] retrieving image: registry.k8s.io/coredns:1.6.2
I0108 19:22:40.026708 90261 image.go:134] retrieving image: registry.k8s.io/kube-scheduler:v1.16.0
I0108 19:22:40.027003 90261 image.go:134] retrieving image: registry.k8s.io/pause:3.1
I0108 19:22:40.032619 90261 image.go:177] daemon lookup for registry.k8s.io/pause:3.1: Error response from daemon: No such image: registry.k8s.io/pause:3.1
I0108 19:22:40.032635 90261 image.go:177] daemon lookup for registry.k8s.io/kube-scheduler:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-scheduler:v1.16.0
I0108 19:22:40.032637 90261 image.go:177] daemon lookup for gcr.io/k8s-minikube/storage-provisioner:v5: Error response from daemon: No such image: gcr.io/k8s-minikube/storage-provisioner:v5
I0108 19:22:40.034040 90261 image.go:177] daemon lookup for registry.k8s.io/etcd:3.3.15-0: Error response from daemon: No such image: registry.k8s.io/etcd:3.3.15-0
I0108 19:22:40.033967 90261 image.go:177] daemon lookup for registry.k8s.io/kube-apiserver:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-apiserver:v1.16.0
I0108 19:22:40.034158 90261 image.go:177] daemon lookup for registry.k8s.io/kube-controller-manager:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-controller-manager:v1.16.0
I0108 19:22:40.034638 90261 image.go:177] daemon lookup for registry.k8s.io/kube-proxy:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-proxy:v1.16.0
I0108 19:22:40.034802 90261 image.go:177] daemon lookup for registry.k8s.io/coredns:1.6.2: Error response from daemon: No such image: registry.k8s.io/coredns:1.6.2
I0108 19:22:40.441037 90261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/pause:3.1
I0108 19:22:40.459710 90261 cache_images.go:116] "registry.k8s.io/pause:3.1" needs transfer: "registry.k8s.io/pause:3.1" does not exist at hash "da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e" in container runtime
I0108 19:22:40.459773 90261 docker.go:323] Removing image: registry.k8s.io/pause:3.1
I0108 19:22:40.459832 90261 ssh_runner.go:195] Run: docker rmi registry.k8s.io/pause:3.1
I0108 19:22:40.465132 90261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-scheduler:v1.16.0
I0108 19:22:40.480336 90261 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17866-74927/.minikube/cache/images/amd64/registry.k8s.io/pause_3.1
I0108 19:22:40.486605 90261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/etcd:3.3.15-0
I0108 19:22:40.488102 90261 cache_images.go:116] "registry.k8s.io/kube-scheduler:v1.16.0" needs transfer: "registry.k8s.io/kube-scheduler:v1.16.0" does not exist at hash "301ddc62b80b16315d3c2653cf3888370394277afb3187614cfa20edc352ca0a" in container runtime
I0108 19:22:40.488140 90261 docker.go:323] Removing image: registry.k8s.io/kube-scheduler:v1.16.0
I0108 19:22:40.488195 90261 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-scheduler:v1.16.0
I0108 19:22:40.506704 90261 cache_images.go:116] "registry.k8s.io/etcd:3.3.15-0" needs transfer: "registry.k8s.io/etcd:3.3.15-0" does not exist at hash "b2756210eeabf84f3221da9959e9483f3919dc2aaab4cd45e7cd072fcbde27ed" in container runtime
I0108 19:22:40.506744 90261 docker.go:323] Removing image: registry.k8s.io/etcd:3.3.15-0
I0108 19:22:40.506826 90261 ssh_runner.go:195] Run: docker rmi registry.k8s.io/etcd:3.3.15-0
I0108 19:22:40.508852 90261 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17866-74927/.minikube/cache/images/amd64/registry.k8s.io/kube-scheduler_v1.16.0
I0108 19:22:40.526332 90261 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17866-74927/.minikube/cache/images/amd64/registry.k8s.io/etcd_3.3.15-0
I0108 19:22:40.566797 90261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-controller-manager:v1.16.0
I0108 19:22:40.587067 90261 cache_images.go:116] "registry.k8s.io/kube-controller-manager:v1.16.0" needs transfer: "registry.k8s.io/kube-controller-manager:v1.16.0" does not exist at hash "06a629a7e51cdcc81a5ed6a3e6650348312f20c954ac52ee489a023628ec9c7d" in container runtime
I0108 19:22:40.587098 90261 docker.go:323] Removing image: registry.k8s.io/kube-controller-manager:v1.16.0
I0108 19:22:40.587172 90261 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-controller-manager:v1.16.0
I0108 19:22:40.592555 90261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-apiserver:v1.16.0
I0108 19:22:40.606592 90261 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17866-74927/.minikube/cache/images/amd64/registry.k8s.io/kube-controller-manager_v1.16.0
I0108 19:22:40.612067 90261 cache_images.go:116] "registry.k8s.io/kube-apiserver:v1.16.0" needs transfer: "registry.k8s.io/kube-apiserver:v1.16.0" does not exist at hash "b305571ca60a5a7818bda47da122683d75e8a1907475681ee8b1efbd06bff12e" in container runtime
I0108 19:22:40.612098 90261 docker.go:323] Removing image: registry.k8s.io/kube-apiserver:v1.16.0
I0108 19:22:40.612158 90261 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-apiserver:v1.16.0
I0108 19:22:40.629329 90261 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17866-74927/.minikube/cache/images/amd64/registry.k8s.io/kube-apiserver_v1.16.0
I0108 19:22:40.659412 90261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-proxy:v1.16.0
I0108 19:22:40.678883 90261 cache_images.go:116] "registry.k8s.io/kube-proxy:v1.16.0" needs transfer: "registry.k8s.io/kube-proxy:v1.16.0" does not exist at hash "c21b0c7400f988db4777858edd13b6d3930d62d7ccf026d2415485a52037f384" in container runtime
I0108 19:22:40.678909 90261 docker.go:323] Removing image: registry.k8s.io/kube-proxy:v1.16.0
I0108 19:22:40.678976 90261 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-proxy:v1.16.0
I0108 19:22:40.697215 90261 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17866-74927/.minikube/cache/images/amd64/registry.k8s.io/kube-proxy_v1.16.0
I0108 19:22:40.765414 90261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/coredns:1.6.2
I0108 19:22:40.779755 90261 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} gcr.io/k8s-minikube/storage-provisioner:v5
I0108 19:22:40.784934 90261 cache_images.go:116] "registry.k8s.io/coredns:1.6.2" needs transfer: "registry.k8s.io/coredns:1.6.2" does not exist at hash "bf261d157914477ee1a5969d28ec687f3fbfc9fb5a664b22df78e57023b0e03b" in container runtime
I0108 19:22:40.784960 90261 docker.go:323] Removing image: registry.k8s.io/coredns:1.6.2
I0108 19:22:40.785029 90261 ssh_runner.go:195] Run: docker rmi registry.k8s.io/coredns:1.6.2
I0108 19:22:40.804576 90261 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17866-74927/.minikube/cache/images/amd64/registry.k8s.io/coredns_1.6.2
I0108 19:22:40.804625 90261 cache_images.go:92] LoadImages completed in 783.824842ms
W0108 19:22:40.804681 90261 out.go:239] X Unable to load cached images: loading cached images: stat /Users/jenkins/minikube-integration/17866-74927/.minikube/cache/images/amd64/registry.k8s.io/pause_3.1: no such file or directory
X Unable to load cached images: loading cached images: stat /Users/jenkins/minikube-integration/17866-74927/.minikube/cache/images/amd64/registry.k8s.io/pause_3.1: no such file or directory
I0108 19:22:40.804754 90261 ssh_runner.go:195] Run: docker info --format {{.CgroupDriver}}
I0108 19:22:40.853018 90261 cni.go:84] Creating CNI manager for ""
I0108 19:22:40.853035 90261 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0108 19:22:40.853059 90261 kubeadm.go:87] Using pod CIDR: 10.244.0.0/16
I0108 19:22:40.853076 90261 kubeadm.go:176] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.85.2 APIServerPort:8443 KubernetesVersion:v1.16.0 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:old-k8s-version-901000 NodeName:old-k8s-version-901000 DNSDomain:cluster.local CRISocket:/var/run/dockershim.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.85.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.85.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticP
odPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[hairpinMode:hairpin-veth runtimeRequestTimeout:15m] PrependCriSocketUnix:false}
I0108 19:22:40.853171 90261 kubeadm.go:181] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.85.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: "old-k8s-version-901000"
kubeletExtraArgs:
node-ip: 192.168.85.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.85.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: old-k8s-version-901000
controlPlaneEndpoint: control-plane.minikube.internal:8443
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
listen-metrics-urls: http://127.0.0.1:2381,http://192.168.85.2:2381
kubernetesVersion: v1.16.0
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0108 19:22:40.853223 90261 kubeadm.go:976] kubelet [Unit]
Wants=docker.socket
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.16.0/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=old-k8s-version-901000 --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.85.2
[Install]
config:
{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-901000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:}
I0108 19:22:40.853285 90261 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.16.0
I0108 19:22:40.861915 90261 binaries.go:44] Found k8s binaries, skipping transfer
I0108 19:22:40.861967 90261 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0108 19:22:40.870439 90261 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (348 bytes)
I0108 19:22:40.885939 90261 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0108 19:22:40.901497 90261 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2174 bytes)
I0108 19:22:40.917307 90261 ssh_runner.go:195] Run: grep 192.168.85.2 control-plane.minikube.internal$ /etc/hosts
I0108 19:22:40.921479 90261 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.85.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0108 19:22:40.932137 90261 certs.go:56] Setting up /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000 for IP: 192.168.85.2
I0108 19:22:40.932157 90261 certs.go:190] acquiring lock for shared ca certs: {Name:mk44dcbca6ce5cf77b3bf5ce2248b699d6553e48 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0108 19:22:40.932351 90261 certs.go:199] skipping minikubeCA CA generation: /Users/jenkins/minikube-integration/17866-74927/.minikube/ca.key
I0108 19:22:40.932442 90261 certs.go:199] skipping proxyClientCA CA generation: /Users/jenkins/minikube-integration/17866-74927/.minikube/proxy-client-ca.key
I0108 19:22:40.932493 90261 certs.go:319] generating minikube-user signed cert: /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/client.key
I0108 19:22:40.932506 90261 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/client.crt with IP's: []
I0108 19:22:41.034299 90261 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/client.crt ...
I0108 19:22:41.034317 90261 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/client.crt: {Name:mk24eabae0a29cb5887008a89084d1c1d755d51d Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0108 19:22:41.034689 90261 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/client.key ...
I0108 19:22:41.034700 90261 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/client.key: {Name:mk943257cae6de7d470fe92ce94386415a15a8f9 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0108 19:22:41.034944 90261 certs.go:319] generating minikube signed cert: /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/apiserver.key.43b9df8c
I0108 19:22:41.034958 90261 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/apiserver.crt.43b9df8c with IP's: [192.168.85.2 10.96.0.1 127.0.0.1 10.0.0.1]
I0108 19:22:41.158656 90261 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/apiserver.crt.43b9df8c ...
I0108 19:22:41.158678 90261 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/apiserver.crt.43b9df8c: {Name:mk3a9f5dc852e9dd2c6775f84cc4335888943edf Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0108 19:22:41.158994 90261 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/apiserver.key.43b9df8c ...
I0108 19:22:41.159005 90261 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/apiserver.key.43b9df8c: {Name:mkfbe71ab17b2d996bcacf2a33994804a162980d Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0108 19:22:41.159208 90261 certs.go:337] copying /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/apiserver.crt.43b9df8c -> /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/apiserver.crt
I0108 19:22:41.159397 90261 certs.go:341] copying /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/apiserver.key.43b9df8c -> /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/apiserver.key
I0108 19:22:41.159574 90261 certs.go:319] generating aggregator signed cert: /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/proxy-client.key
I0108 19:22:41.159590 90261 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/proxy-client.crt with IP's: []
I0108 19:22:41.381399 90261 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/proxy-client.crt ...
I0108 19:22:41.381411 90261 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/proxy-client.crt: {Name:mkbb92511f1b368a85dc132abf0f034d3a9d7e0b Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0108 19:22:41.381714 90261 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/proxy-client.key ...
I0108 19:22:41.381723 90261 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/proxy-client.key: {Name:mk1f8b1cfb399fdd9a7556519b654b27742d267b Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0108 19:22:41.382128 90261 certs.go:437] found cert: /Users/jenkins/minikube-integration/17866-74927/.minikube/certs/Users/jenkins/minikube-integration/17866-74927/.minikube/certs/75369.pem (1338 bytes)
W0108 19:22:41.382179 90261 certs.go:433] ignoring /Users/jenkins/minikube-integration/17866-74927/.minikube/certs/Users/jenkins/minikube-integration/17866-74927/.minikube/certs/75369_empty.pem, impossibly tiny 0 bytes
I0108 19:22:41.382192 90261 certs.go:437] found cert: /Users/jenkins/minikube-integration/17866-74927/.minikube/certs/Users/jenkins/minikube-integration/17866-74927/.minikube/certs/ca-key.pem (1675 bytes)
I0108 19:22:41.382224 90261 certs.go:437] found cert: /Users/jenkins/minikube-integration/17866-74927/.minikube/certs/Users/jenkins/minikube-integration/17866-74927/.minikube/certs/ca.pem (1078 bytes)
I0108 19:22:41.382255 90261 certs.go:437] found cert: /Users/jenkins/minikube-integration/17866-74927/.minikube/certs/Users/jenkins/minikube-integration/17866-74927/.minikube/certs/cert.pem (1123 bytes)
I0108 19:22:41.382284 90261 certs.go:437] found cert: /Users/jenkins/minikube-integration/17866-74927/.minikube/certs/Users/jenkins/minikube-integration/17866-74927/.minikube/certs/key.pem (1679 bytes)
I0108 19:22:41.382345 90261 certs.go:437] found cert: /Users/jenkins/minikube-integration/17866-74927/.minikube/files/etc/ssl/certs/Users/jenkins/minikube-integration/17866-74927/.minikube/files/etc/ssl/certs/753692.pem (1708 bytes)
I0108 19:22:41.382866 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1399 bytes)
I0108 19:22:41.403589 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1675 bytes)
I0108 19:22:41.423991 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0108 19:22:41.444513 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/old-k8s-version-901000/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1675 bytes)
I0108 19:22:41.465325 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0108 19:22:41.485762 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1679 bytes)
I0108 19:22:41.506509 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0108 19:22:41.527108 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1675 bytes)
I0108 19:22:41.548427 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/certs/75369.pem --> /usr/share/ca-certificates/75369.pem (1338 bytes)
I0108 19:22:41.570446 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/files/etc/ssl/certs/753692.pem --> /usr/share/ca-certificates/753692.pem (1708 bytes)
I0108 19:22:41.593567 90261 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17866-74927/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0108 19:22:41.614544 90261 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0108 19:22:41.630620 90261 ssh_runner.go:195] Run: openssl version
I0108 19:22:41.636409 90261 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/75369.pem && ln -fs /usr/share/ca-certificates/75369.pem /etc/ssl/certs/75369.pem"
I0108 19:22:41.645616 90261 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/75369.pem
I0108 19:22:41.649881 90261 certs.go:480] hashing: -rw-r--r-- 1 root root 1338 Jan 9 02:38 /usr/share/ca-certificates/75369.pem
I0108 19:22:41.649933 90261 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/75369.pem
I0108 19:22:41.656596 90261 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/75369.pem /etc/ssl/certs/51391683.0"
I0108 19:22:41.665524 90261 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/753692.pem && ln -fs /usr/share/ca-certificates/753692.pem /etc/ssl/certs/753692.pem"
I0108 19:22:41.674643 90261 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/753692.pem
I0108 19:22:41.678774 90261 certs.go:480] hashing: -rw-r--r-- 1 root root 1708 Jan 9 02:38 /usr/share/ca-certificates/753692.pem
I0108 19:22:41.678818 90261 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/753692.pem
I0108 19:22:41.685269 90261 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/753692.pem /etc/ssl/certs/3ec20f2e.0"
I0108 19:22:41.694309 90261 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0108 19:22:41.703177 90261 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0108 19:22:41.707294 90261 certs.go:480] hashing: -rw-r--r-- 1 root root 1111 Jan 9 02:33 /usr/share/ca-certificates/minikubeCA.pem
I0108 19:22:41.707354 90261 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0108 19:22:41.714112 90261 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0108 19:22:41.723422 90261 ssh_runner.go:195] Run: ls /var/lib/minikube/certs/etcd
I0108 19:22:41.727637 90261 certs.go:353] certs directory doesn't exist, likely first start: ls /var/lib/minikube/certs/etcd: Process exited with status 2
stdout:
stderr:
ls: cannot access '/var/lib/minikube/certs/etcd': No such file or directory
I0108 19:22:41.727693 90261 kubeadm.go:404] StartCluster: {Name:old-k8s-version-901000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-901000 Namespace:default APIServerName:minikubeCA APISer
verNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.85.2 Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false Disab
leMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs:}
I0108 19:22:41.727827 90261 ssh_runner.go:195] Run: docker ps --filter status=paused --filter=name=k8s_.*_(kube-system)_ --format={{.ID}}
I0108 19:22:41.748902 90261 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0108 19:22:41.757828 90261 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0108 19:22:41.766620 90261 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0108 19:22:41.766674 90261 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0108 19:22:41.774862 90261 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0108 19:22:41.774898 90261 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0108 19:22:41.852969 90261 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0108 19:22:41.853008 90261 kubeadm.go:322] [preflight] Running pre-flight checks
I0108 19:22:42.106001 90261 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0108 19:22:42.106087 90261 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0108 19:22:42.106170 90261 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0108 19:22:42.273462 90261 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0108 19:22:42.274393 90261 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0108 19:22:42.280195 90261 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0108 19:22:42.347672 90261 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0108 19:22:42.369322 90261 out.go:204] - Generating certificates and keys ...
I0108 19:22:42.369403 90261 kubeadm.go:322] [certs] Using existing ca certificate authority
I0108 19:22:42.369464 90261 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0108 19:22:42.455196 90261 kubeadm.go:322] [certs] Generating "apiserver-kubelet-client" certificate and key
I0108 19:22:42.516790 90261 kubeadm.go:322] [certs] Generating "front-proxy-ca" certificate and key
I0108 19:22:42.603949 90261 kubeadm.go:322] [certs] Generating "front-proxy-client" certificate and key
I0108 19:22:42.736117 90261 kubeadm.go:322] [certs] Generating "etcd/ca" certificate and key
I0108 19:22:42.895025 90261 kubeadm.go:322] [certs] Generating "etcd/server" certificate and key
I0108 19:22:42.895140 90261 kubeadm.go:322] [certs] etcd/server serving cert is signed for DNS names [old-k8s-version-901000 localhost] and IPs [192.168.85.2 127.0.0.1 ::1]
I0108 19:22:43.066552 90261 kubeadm.go:322] [certs] Generating "etcd/peer" certificate and key
I0108 19:22:43.066665 90261 kubeadm.go:322] [certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-901000 localhost] and IPs [192.168.85.2 127.0.0.1 ::1]
I0108 19:22:43.226978 90261 kubeadm.go:322] [certs] Generating "etcd/healthcheck-client" certificate and key
I0108 19:22:43.402565 90261 kubeadm.go:322] [certs] Generating "apiserver-etcd-client" certificate and key
I0108 19:22:43.607173 90261 kubeadm.go:322] [certs] Generating "sa" key and public key
I0108 19:22:43.607228 90261 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0108 19:22:43.678519 90261 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0108 19:22:43.863740 90261 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0108 19:22:44.112545 90261 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0108 19:22:44.319139 90261 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0108 19:22:44.319618 90261 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0108 19:22:44.342751 90261 out.go:204] - Booting up control plane ...
I0108 19:22:44.342888 90261 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0108 19:22:44.343006 90261 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0108 19:22:44.343125 90261 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0108 19:22:44.343279 90261 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0108 19:22:44.343518 90261 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0108 19:23:24.328211 90261 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0108 19:23:24.328990 90261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0108 19:23:24.329199 90261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0108 19:23:29.329523 90261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0108 19:23:29.329701 90261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0108 19:23:39.330084 90261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0108 19:23:39.330264 90261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0108 19:23:59.331588 90261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0108 19:23:59.331857 90261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0108 19:24:39.332272 90261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0108 19:24:39.332510 90261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0108 19:24:39.332524 90261 kubeadm.go:322]
I0108 19:24:39.332561 90261 kubeadm.go:322] Unfortunately, an error has occurred:
I0108 19:24:39.332606 90261 kubeadm.go:322] timed out waiting for the condition
I0108 19:24:39.332614 90261 kubeadm.go:322]
I0108 19:24:39.332647 90261 kubeadm.go:322] This error is likely caused by:
I0108 19:24:39.332678 90261 kubeadm.go:322] - The kubelet is not running
I0108 19:24:39.332837 90261 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0108 19:24:39.332852 90261 kubeadm.go:322]
I0108 19:24:39.332967 90261 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0108 19:24:39.333001 90261 kubeadm.go:322] - 'systemctl status kubelet'
I0108 19:24:39.333034 90261 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0108 19:24:39.333043 90261 kubeadm.go:322]
I0108 19:24:39.333182 90261 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0108 19:24:39.333285 90261 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0108 19:24:39.333377 90261 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0108 19:24:39.333438 90261 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0108 19:24:39.333514 90261 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0108 19:24:39.333553 90261 kubeadm.go:322] - 'docker logs CONTAINERID'
I0108 19:24:39.335204 90261 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0108 19:24:39.335278 90261 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0108 19:24:39.335398 90261 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
I0108 19:24:39.335482 90261 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0108 19:24:39.335577 90261 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0108 19:24:39.335634 90261 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
W0108 19:24:39.335755 90261 out.go:239] ! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-901000 localhost] and IPs [192.168.85.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-901000 localhost] and IPs [192.168.85.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-901000 localhost] and IPs [192.168.85.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-901000 localhost] and IPs [192.168.85.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
I0108 19:24:39.335793 90261 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm reset --cri-socket /var/run/dockershim.sock --force"
I0108 19:24:39.743634 90261 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service kubelet
I0108 19:24:39.754374 90261 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0108 19:24:39.754436 90261 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0108 19:24:39.762542 90261 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0108 19:24:39.762564 90261 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0108 19:24:39.817331 90261 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0108 19:24:39.817374 90261 kubeadm.go:322] [preflight] Running pre-flight checks
I0108 19:24:40.052781 90261 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0108 19:24:40.052874 90261 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0108 19:24:40.052957 90261 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0108 19:24:40.222533 90261 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0108 19:24:40.223470 90261 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0108 19:24:40.229493 90261 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0108 19:24:40.306605 90261 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0108 19:24:40.327951 90261 out.go:204] - Generating certificates and keys ...
I0108 19:24:40.328012 90261 kubeadm.go:322] [certs] Using existing ca certificate authority
I0108 19:24:40.328078 90261 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0108 19:24:40.328143 90261 kubeadm.go:322] [certs] Using existing apiserver-kubelet-client certificate and key on disk
I0108 19:24:40.328201 90261 kubeadm.go:322] [certs] Using existing front-proxy-ca certificate authority
I0108 19:24:40.328273 90261 kubeadm.go:322] [certs] Using existing front-proxy-client certificate and key on disk
I0108 19:24:40.328325 90261 kubeadm.go:322] [certs] Using existing etcd/ca certificate authority
I0108 19:24:40.328371 90261 kubeadm.go:322] [certs] Using existing etcd/server certificate and key on disk
I0108 19:24:40.328411 90261 kubeadm.go:322] [certs] Using existing etcd/peer certificate and key on disk
I0108 19:24:40.328471 90261 kubeadm.go:322] [certs] Using existing etcd/healthcheck-client certificate and key on disk
I0108 19:24:40.328534 90261 kubeadm.go:322] [certs] Using existing apiserver-etcd-client certificate and key on disk
I0108 19:24:40.328570 90261 kubeadm.go:322] [certs] Using the existing "sa" key
I0108 19:24:40.328611 90261 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0108 19:24:40.485049 90261 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0108 19:24:40.553279 90261 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0108 19:24:40.616243 90261 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0108 19:24:40.893148 90261 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0108 19:24:40.893653 90261 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0108 19:24:40.915042 90261 out.go:204] - Booting up control plane ...
I0108 19:24:40.915170 90261 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0108 19:24:40.915316 90261 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0108 19:24:40.915442 90261 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0108 19:24:40.915577 90261 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0108 19:24:40.915867 90261 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0108 19:25:20.901459 90261 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0108 19:25:20.902173 90261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0108 19:25:20.902329 90261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0108 19:25:25.903233 90261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0108 19:25:25.903509 90261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0108 19:25:35.904481 90261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0108 19:25:35.904721 90261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0108 19:25:55.905403 90261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0108 19:25:55.905621 90261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0108 19:26:35.905184 90261 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0108 19:26:35.905403 90261 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0108 19:26:35.905416 90261 kubeadm.go:322]
I0108 19:26:35.905454 90261 kubeadm.go:322] Unfortunately, an error has occurred:
I0108 19:26:35.905486 90261 kubeadm.go:322] timed out waiting for the condition
I0108 19:26:35.905491 90261 kubeadm.go:322]
I0108 19:26:35.905524 90261 kubeadm.go:322] This error is likely caused by:
I0108 19:26:35.905550 90261 kubeadm.go:322] - The kubelet is not running
I0108 19:26:35.905640 90261 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0108 19:26:35.905651 90261 kubeadm.go:322]
I0108 19:26:35.905737 90261 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0108 19:26:35.905769 90261 kubeadm.go:322] - 'systemctl status kubelet'
I0108 19:26:35.905804 90261 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0108 19:26:35.905810 90261 kubeadm.go:322]
I0108 19:26:35.905894 90261 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0108 19:26:35.905971 90261 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0108 19:26:35.906036 90261 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0108 19:26:35.906080 90261 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0108 19:26:35.906175 90261 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0108 19:26:35.906218 90261 kubeadm.go:322] - 'docker logs CONTAINERID'
I0108 19:26:35.907437 90261 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0108 19:26:35.907496 90261 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0108 19:26:35.907623 90261 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
I0108 19:26:35.907698 90261 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0108 19:26:35.907759 90261 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0108 19:26:35.907843 90261 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
I0108 19:26:35.907868 90261 kubeadm.go:406] StartCluster complete in 3m54.185327438s
I0108 19:26:35.907959 90261 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-apiserver --format={{.ID}}
I0108 19:26:35.927262 90261 logs.go:284] 0 containers: []
W0108 19:26:35.927276 90261 logs.go:286] No container was found matching "kube-apiserver"
I0108 19:26:35.927358 90261 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_etcd --format={{.ID}}
I0108 19:26:35.946472 90261 logs.go:284] 0 containers: []
W0108 19:26:35.946489 90261 logs.go:286] No container was found matching "etcd"
I0108 19:26:35.946562 90261 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_coredns --format={{.ID}}
I0108 19:26:35.965846 90261 logs.go:284] 0 containers: []
W0108 19:26:35.965860 90261 logs.go:286] No container was found matching "coredns"
I0108 19:26:35.965928 90261 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-scheduler --format={{.ID}}
I0108 19:26:35.982876 90261 logs.go:284] 0 containers: []
W0108 19:26:35.982891 90261 logs.go:286] No container was found matching "kube-scheduler"
I0108 19:26:35.982959 90261 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-proxy --format={{.ID}}
I0108 19:26:36.000792 90261 logs.go:284] 0 containers: []
W0108 19:26:36.000806 90261 logs.go:286] No container was found matching "kube-proxy"
I0108 19:26:36.000874 90261 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-controller-manager --format={{.ID}}
I0108 19:26:36.018260 90261 logs.go:284] 0 containers: []
W0108 19:26:36.018276 90261 logs.go:286] No container was found matching "kube-controller-manager"
I0108 19:26:36.018352 90261 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kindnet --format={{.ID}}
I0108 19:26:36.035907 90261 logs.go:284] 0 containers: []
W0108 19:26:36.035921 90261 logs.go:286] No container was found matching "kindnet"
I0108 19:26:36.035928 90261 logs.go:123] Gathering logs for kubelet ...
I0108 19:26:36.035935 90261 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
I0108 19:26:36.069951 90261 logs.go:123] Gathering logs for dmesg ...
I0108 19:26:36.069966 90261 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0108 19:26:36.082201 90261 logs.go:123] Gathering logs for describe nodes ...
I0108 19:26:36.082217 90261 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0108 19:26:36.133173 90261 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0108 19:26:36.133190 90261 logs.go:123] Gathering logs for Docker ...
I0108 19:26:36.133206 90261 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u docker -u cri-docker -n 400"
I0108 19:26:36.148420 90261 logs.go:123] Gathering logs for container status ...
I0108 19:26:36.148435 90261 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
W0108 19:26:36.196097 90261 out.go:369] Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0108 19:26:36.196122 90261 out.go:239] *
*
W0108 19:26:36.196176 90261 out.go:239] X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0108 19:26:36.196205 90261 out.go:239] *
*
W0108 19:26:36.196810 90261 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0108 19:26:36.259204 90261 out.go:177]
W0108 19:26:36.301452 90261 out.go:239] X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0108 19:26:36.301515 90261 out.go:239] * Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
* Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
W0108 19:26:36.301541 90261 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/4172
* Related issue: https://github.com/kubernetes/minikube/issues/4172
I0108 19:26:36.343480 90261 out.go:177]
** /stderr **
start_stop_delete_test.go:188: failed starting minikube -first start-. args "out/minikube-darwin-amd64 start -p old-k8s-version-901000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0": exit status 109
helpers_test.go:222: -----------------------post-mortem--------------------------------
helpers_test.go:230: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/FirstStart]: docker inspect <======
helpers_test.go:231: (dbg) Run: docker inspect old-k8s-version-901000
helpers_test.go:235: (dbg) docker inspect old-k8s-version-901000:
-- stdout --
[
{
"Id": "aa25a1062c3637249ed98695e91a47377ce4dea8ff0830e7d303498491e8835c",
"Created": "2024-01-09T03:22:27.685275696Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 294952,
"ExitCode": 0,
"Error": "",
"StartedAt": "2024-01-09T03:22:27.950231376Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:9941de2e064a4a6a7155bfc66cedd2854b8c725b77bb8d4eaf81bef39f951dd7",
"ResolvConfPath": "/var/lib/docker/containers/aa25a1062c3637249ed98695e91a47377ce4dea8ff0830e7d303498491e8835c/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/aa25a1062c3637249ed98695e91a47377ce4dea8ff0830e7d303498491e8835c/hostname",
"HostsPath": "/var/lib/docker/containers/aa25a1062c3637249ed98695e91a47377ce4dea8ff0830e7d303498491e8835c/hosts",
"LogPath": "/var/lib/docker/containers/aa25a1062c3637249ed98695e91a47377ce4dea8ff0830e7d303498491e8835c/aa25a1062c3637249ed98695e91a47377ce4dea8ff0830e7d303498491e8835c-json.log",
"Name": "/old-k8s-version-901000",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro",
"old-k8s-version-901000:/var"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "old-k8s-version-901000",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 2306867200,
"NanoCpus": 2000000000,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": [],
"BlkioDeviceWriteBps": [],
"BlkioDeviceReadIOps": [],
"BlkioDeviceWriteIOps": [],
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 2306867200,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/511bcc6ca847d5924ffe005728f7ed77c8094a92053674c964ad837af718051a-init/diff:/var/lib/docker/overlay2/60277c56cb2e84cbe47fd8ed3c79b85a017889e24b19778a8fc4b14c01478988/diff",
"MergedDir": "/var/lib/docker/overlay2/511bcc6ca847d5924ffe005728f7ed77c8094a92053674c964ad837af718051a/merged",
"UpperDir": "/var/lib/docker/overlay2/511bcc6ca847d5924ffe005728f7ed77c8094a92053674c964ad837af718051a/diff",
"WorkDir": "/var/lib/docker/overlay2/511bcc6ca847d5924ffe005728f7ed77c8094a92053674c964ad837af718051a/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "old-k8s-version-901000",
"Source": "/var/lib/docker/volumes/old-k8s-version-901000/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "old-k8s-version-901000",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"22/tcp": {},
"2376/tcp": {},
"32443/tcp": {},
"5000/tcp": {},
"8443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "old-k8s-version-901000",
"name.minikube.sigs.k8s.io": "old-k8s-version-901000",
"role.minikube.sigs.k8s.io": ""
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "65c44b0e10ad32eff889c1f6bb439c21ce1470f12efaf7b43221a9d5ad40761a",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "49929"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "49930"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "49931"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "49932"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "49928"
}
]
},
"SandboxKey": "/var/run/docker/netns/65c44b0e10ad",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"old-k8s-version-901000": {
"IPAMConfig": {
"IPv4Address": "192.168.85.2"
},
"Links": null,
"Aliases": [
"aa25a1062c36",
"old-k8s-version-901000"
],
"MacAddress": "02:42:c0:a8:55:02",
"NetworkID": "876d1b6b5bcffa0a183a1c34f9924af9d72a7d63d67d2b9f07e88b4f08db4216",
"EndpointID": "c677f4f15eee1cfda0bebd0c72a3f693127afc92899807728b1626f9989feef9",
"Gateway": "192.168.85.1",
"IPAddress": "192.168.85.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"DriverOpts": null
}
}
}
}
]
-- /stdout --
helpers_test.go:239: (dbg) Run: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-901000 -n old-k8s-version-901000
E0108 19:26:36.502191 75369 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17866-74927/.minikube/profiles/calico-798000/client.crt: no such file or directory
helpers_test.go:239: (dbg) Non-zero exit: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-901000 -n old-k8s-version-901000: exit status 6 (385.312283ms)
-- stdout --
Running
WARNING: Your kubectl is pointing to stale minikube-vm.
To fix the kubectl context, run `minikube update-context`
-- /stdout --
** stderr **
E0108 19:26:36.872500 91042 status.go:415] kubeconfig endpoint: extract IP: "old-k8s-version-901000" does not appear in /Users/jenkins/minikube-integration/17866-74927/kubeconfig
** /stderr **
helpers_test.go:239: status error: exit status 6 (may be ok)
helpers_test.go:241: "old-k8s-version-901000" host is not running, skipping log retrieval (state="Running\nWARNING: Your kubectl is pointing to stale minikube-vm.\nTo fix the kubectl context, run `minikube update-context`")
--- FAIL: TestStartStop/group/old-k8s-version/serial/FirstStart (253.56s)