=== RUN TestStartStop/group/old-k8s-version/serial/FirstStart
start_stop_delete_test.go:186: (dbg) Run: out/minikube-darwin-amd64 start -p old-k8s-version-187000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0
start_stop_delete_test.go:186: (dbg) Non-zero exit: out/minikube-darwin-amd64 start -p old-k8s-version-187000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0: exit status 109 (4m16.928427062s)
-- stdout --
* [old-k8s-version-187000] minikube v1.32.0 on Darwin 14.3.1
- MINIKUBE_LOCATION=18165
- KUBECONFIG=/Users/jenkins/minikube-integration/18165-38421/kubeconfig
- MINIKUBE_BIN=out/minikube-darwin-amd64
- MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
- MINIKUBE_HOME=/Users/jenkins/minikube-integration/18165-38421/.minikube
- MINIKUBE_FORCE_SYSTEMD=
* Using the docker driver based on user configuration
* Using Docker Desktop driver with root privileges
* Starting control plane node old-k8s-version-187000 in cluster old-k8s-version-187000
* Pulling base image v0.0.42-1704759386-17866 ...
* Creating docker container (CPUs=2, Memory=2200MB) ...
* Preparing Kubernetes v1.16.0 on Docker 24.0.7 ...
- Generating certificates and keys ...
- Booting up control plane ...
- Generating certificates and keys ...
- Booting up control plane ...
-- /stdout --
** stderr **
I0213 19:12:00.561822 55174 out.go:291] Setting OutFile to fd 1 ...
I0213 19:12:00.562223 55174 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0213 19:12:00.562236 55174 out.go:304] Setting ErrFile to fd 2...
I0213 19:12:00.562244 55174 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0213 19:12:00.562538 55174 root.go:338] Updating PATH: /Users/jenkins/minikube-integration/18165-38421/.minikube/bin
I0213 19:12:00.564779 55174 out.go:298] Setting JSON to false
I0213 19:12:00.588457 55174 start.go:128] hostinfo: {"hostname":"MacOS-Agent-1.local","uptime":17179,"bootTime":1707863141,"procs":514,"os":"darwin","platform":"darwin","platformFamily":"Standalone Workstation","platformVersion":"14.3.1","kernelVersion":"23.3.0","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"b7610dcb-1435-5842-8d5a-b2388403fea3"}
W0213 19:12:00.588569 55174 start.go:136] gopshost.Virtualization returned error: not implemented yet
I0213 19:12:00.610637 55174 out.go:177] * [old-k8s-version-187000] minikube v1.32.0 on Darwin 14.3.1
I0213 19:12:00.673359 55174 out.go:177] - MINIKUBE_LOCATION=18165
I0213 19:12:00.652499 55174 notify.go:220] Checking for updates...
I0213 19:12:00.715417 55174 out.go:177] - KUBECONFIG=/Users/jenkins/minikube-integration/18165-38421/kubeconfig
I0213 19:12:00.757224 55174 out.go:177] - MINIKUBE_BIN=out/minikube-darwin-amd64
I0213 19:12:00.778448 55174 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0213 19:12:00.820111 55174 out.go:177] - MINIKUBE_HOME=/Users/jenkins/minikube-integration/18165-38421/.minikube
I0213 19:12:00.878377 55174 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0213 19:12:00.899704 55174 config.go:182] Loaded profile config "kubenet-210000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.28.4
I0213 19:12:00.899797 55174 driver.go:392] Setting default libvirt URI to qemu:///system
I0213 19:12:00.958774 55174 docker.go:122] docker version: linux-25.0.3:Docker Desktop 4.27.2 (137060)
I0213 19:12:00.958932 55174 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0213 19:12:01.081139 55174 info.go:266] docker info: {ID:37b3081a-8cd6-457e-b2a4-79dc82345b06 Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:73 OomKillDisable:false NGoroutines:115 SystemTime:2024-02-14 03:12:01.064951267 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:26 KernelVersion:6.6.12-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServerAddress:
https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:12 MemTotal:6213296128 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=unconfined name
=cgroupns] ProductLicense: Warnings:[WARNING: daemon is not using the default seccomp profile] ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1-desktop.4] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.5-desktop.1] map[Name:debug Path:/Users/jenkins/.docker/cli-plugins/docker-debug SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-debug] ShortDescription:Get a shell into any image or container. Vendor:Docker Inc. Version:0.0.24] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker D
ev Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Manages Docker extensions Vendor:Docker Inc. Version:v0.2.21] map[Name:feedback Path:/Users/jenkins/.docker/cli-plugins/docker-feedback SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-feedback] ShortDescription:Provide feedback, right in your terminal! Vendor:Docker Inc. Version:v1.0.4] map[Name:init Path:/Users/jenkins/.docker/cli-plugins/docker-init SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-init] ShortDescription:Creates Docker-related starter files for your project Vendor:Docker Inc. Version:v1.0.0] map[Name:sbom Path:/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM)
for an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Err:failed to fetch metadata: fork/exec /Users/jenkins/.docker/cli-plugins/docker-scan: no such file or directory Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Docker Scout Vendor:Docker Inc. Version:v1.4.1]] Warnings:<nil>}}
I0213 19:12:01.103033 55174 out.go:177] * Using the docker driver based on user configuration
I0213 19:12:01.160777 55174 start.go:298] selected driver: docker
I0213 19:12:01.160790 55174 start.go:902] validating driver "docker" against <nil>
I0213 19:12:01.160799 55174 start.go:913] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0213 19:12:01.164273 55174 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0213 19:12:01.277413 55174 info.go:266] docker info: {ID:37b3081a-8cd6-457e-b2a4-79dc82345b06 Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:73 OomKillDisable:false NGoroutines:115 SystemTime:2024-02-14 03:12:01.267598741 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:26 KernelVersion:6.6.12-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServerAddress:
https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:12 MemTotal:6213296128 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=unconfined name
=cgroupns] ProductLicense: Warnings:[WARNING: daemon is not using the default seccomp profile] ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1-desktop.4] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.5-desktop.1] map[Name:debug Path:/Users/jenkins/.docker/cli-plugins/docker-debug SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-debug] ShortDescription:Get a shell into any image or container. Vendor:Docker Inc. Version:0.0.24] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker D
ev Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Manages Docker extensions Vendor:Docker Inc. Version:v0.2.21] map[Name:feedback Path:/Users/jenkins/.docker/cli-plugins/docker-feedback SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-feedback] ShortDescription:Provide feedback, right in your terminal! Vendor:Docker Inc. Version:v1.0.4] map[Name:init Path:/Users/jenkins/.docker/cli-plugins/docker-init SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-init] ShortDescription:Creates Docker-related starter files for your project Vendor:Docker Inc. Version:v1.0.0] map[Name:sbom Path:/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM)
for an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Err:failed to fetch metadata: fork/exec /Users/jenkins/.docker/cli-plugins/docker-scan: no such file or directory Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Docker Scout Vendor:Docker Inc. Version:v1.4.1]] Warnings:<nil>}}
I0213 19:12:01.277611 55174 start_flags.go:307] no existing cluster config was found, will generate one from the flags
I0213 19:12:01.277810 55174 start_flags.go:927] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0213 19:12:01.299124 55174 out.go:177] * Using Docker Desktop driver with root privileges
I0213 19:12:01.320142 55174 cni.go:84] Creating CNI manager for ""
I0213 19:12:01.320174 55174 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0213 19:12:01.320189 55174 start_flags.go:321] config:
{Name:old-k8s-version-187000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-187000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local Containe
rRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs:}
I0213 19:12:01.341773 55174 out.go:177] * Starting control plane node old-k8s-version-187000 in cluster old-k8s-version-187000
I0213 19:12:01.384005 55174 cache.go:121] Beginning downloading kic base image for docker with docker
I0213 19:12:01.404809 55174 out.go:177] * Pulling base image v0.0.42-1704759386-17866 ...
I0213 19:12:01.462980 55174 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0213 19:12:01.463051 55174 image.go:79] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 in local docker daemon
I0213 19:12:01.463056 55174 preload.go:148] Found local preload: /Users/jenkins/minikube-integration/18165-38421/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4
I0213 19:12:01.463086 55174 cache.go:56] Caching tarball of preloaded images
I0213 19:12:01.463247 55174 preload.go:174] Found /Users/jenkins/minikube-integration/18165-38421/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0213 19:12:01.463261 55174 cache.go:59] Finished verifying existence of preloaded tar for v1.16.0 on docker
I0213 19:12:01.463890 55174 profile.go:148] Saving config to /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/config.json ...
I0213 19:12:01.464067 55174 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/config.json: {Name:mkbc6ac6a942e0a66e41515601eaea1747f53b88 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 19:12:01.516381 55174 image.go:83] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 in local docker daemon, skipping pull
I0213 19:12:01.516418 55174 cache.go:144] gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 exists in daemon, skipping load
I0213 19:12:01.516436 55174 cache.go:194] Successfully downloaded all kic artifacts
I0213 19:12:01.516472 55174 start.go:365] acquiring machines lock for old-k8s-version-187000: {Name:mk0547224fc7a975c28768405bd89305d57998ba Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0213 19:12:01.516605 55174 start.go:369] acquired machines lock for "old-k8s-version-187000" in 120.12µs
I0213 19:12:01.516634 55174 start.go:93] Provisioning new machine with config: &{Name:old-k8s-version-187000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-187000 Namespace:default APIServer
Name:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:f
alse DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs:} &{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}
I0213 19:12:01.516710 55174 start.go:125] createHost starting for "" (driver="docker")
I0213 19:12:01.538019 55174 out.go:204] * Creating docker container (CPUs=2, Memory=2200MB) ...
I0213 19:12:01.538295 55174 start.go:159] libmachine.API.Create for "old-k8s-version-187000" (driver="docker")
I0213 19:12:01.538324 55174 client.go:168] LocalClient.Create starting
I0213 19:12:01.538488 55174 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/18165-38421/.minikube/certs/ca.pem
I0213 19:12:01.538562 55174 main.go:141] libmachine: Decoding PEM data...
I0213 19:12:01.538591 55174 main.go:141] libmachine: Parsing certificate...
I0213 19:12:01.538661 55174 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/18165-38421/.minikube/certs/cert.pem
I0213 19:12:01.538717 55174 main.go:141] libmachine: Decoding PEM data...
I0213 19:12:01.538729 55174 main.go:141] libmachine: Parsing certificate...
I0213 19:12:01.559321 55174 cli_runner.go:164] Run: docker network inspect old-k8s-version-187000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
W0213 19:12:01.610290 55174 cli_runner.go:211] docker network inspect old-k8s-version-187000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1
I0213 19:12:01.610391 55174 network_create.go:281] running [docker network inspect old-k8s-version-187000] to gather additional debugging logs...
I0213 19:12:01.610414 55174 cli_runner.go:164] Run: docker network inspect old-k8s-version-187000
W0213 19:12:01.661115 55174 cli_runner.go:211] docker network inspect old-k8s-version-187000 returned with exit code 1
I0213 19:12:01.661149 55174 network_create.go:284] error running [docker network inspect old-k8s-version-187000]: docker network inspect old-k8s-version-187000: exit status 1
stdout:
[]
stderr:
Error response from daemon: network old-k8s-version-187000 not found
I0213 19:12:01.661165 55174 network_create.go:286] output of [docker network inspect old-k8s-version-187000]: -- stdout --
[]
-- /stdout --
** stderr **
Error response from daemon: network old-k8s-version-187000 not found
** /stderr **
I0213 19:12:01.661304 55174 cli_runner.go:164] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0213 19:12:01.718523 55174 network.go:210] skipping subnet 192.168.49.0/24 that is reserved: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0213 19:12:01.720153 55174 network.go:210] skipping subnet 192.168.58.0/24 that is reserved: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0213 19:12:01.721501 55174 network.go:210] skipping subnet 192.168.67.0/24 that is reserved: &{IP:192.168.67.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.67.0/24 Gateway:192.168.67.1 ClientMin:192.168.67.2 ClientMax:192.168.67.254 Broadcast:192.168.67.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0213 19:12:01.722845 55174 network.go:210] skipping subnet 192.168.76.0/24 that is reserved: &{IP:192.168.76.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.76.0/24 Gateway:192.168.76.1 ClientMin:192.168.76.2 ClientMax:192.168.76.254 Broadcast:192.168.76.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0213 19:12:01.723211 55174 network.go:207] using free private subnet 192.168.85.0/24: &{IP:192.168.85.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.85.0/24 Gateway:192.168.85.1 ClientMin:192.168.85.2 ClientMax:192.168.85.254 Broadcast:192.168.85.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc002414340}
I0213 19:12:01.723227 55174 network_create.go:124] attempt to create docker network old-k8s-version-187000 192.168.85.0/24 with gateway 192.168.85.1 and MTU of 65535 ...
I0213 19:12:01.723294 55174 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.85.0/24 --gateway=192.168.85.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-187000 old-k8s-version-187000
I0213 19:12:01.811031 55174 network_create.go:108] docker network old-k8s-version-187000 192.168.85.0/24 created
I0213 19:12:01.811073 55174 kic.go:121] calculated static IP "192.168.85.2" for the "old-k8s-version-187000" container
I0213 19:12:01.811187 55174 cli_runner.go:164] Run: docker ps -a --format {{.Names}}
I0213 19:12:01.863382 55174 cli_runner.go:164] Run: docker volume create old-k8s-version-187000 --label name.minikube.sigs.k8s.io=old-k8s-version-187000 --label created_by.minikube.sigs.k8s.io=true
I0213 19:12:01.917347 55174 oci.go:103] Successfully created a docker volume old-k8s-version-187000
I0213 19:12:01.917488 55174 cli_runner.go:164] Run: docker run --rm --name old-k8s-version-187000-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-187000 --entrypoint /usr/bin/test -v old-k8s-version-187000:/var gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 -d /var/lib
I0213 19:12:02.309967 55174 oci.go:107] Successfully prepared a docker volume old-k8s-version-187000
I0213 19:12:02.310012 55174 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0213 19:12:02.310027 55174 kic.go:194] Starting extracting preloaded images to volume ...
I0213 19:12:02.310149 55174 cli_runner.go:164] Run: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/18165-38421/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-187000:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 -I lz4 -xf /preloaded.tar -C /extractDir
I0213 19:12:04.415675 55174 cli_runner.go:217] Completed: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/18165-38421/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-187000:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 -I lz4 -xf /preloaded.tar -C /extractDir: (2.105470822s)
I0213 19:12:04.415703 55174 kic.go:203] duration metric: took 2.105690 seconds to extract preloaded images to volume
I0213 19:12:04.415831 55174 cli_runner.go:164] Run: docker info --format "'{{json .SecurityOptions}}'"
I0213 19:12:04.524333 55174 cli_runner.go:164] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname old-k8s-version-187000 --name old-k8s-version-187000 --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-187000 --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=old-k8s-version-187000 --network old-k8s-version-187000 --ip 192.168.85.2 --volume old-k8s-version-187000:/var --security-opt apparmor=unconfined --memory=2200mb --memory-swap=2200mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0
I0213 19:12:04.795219 55174 cli_runner.go:164] Run: docker container inspect old-k8s-version-187000 --format={{.State.Running}}
I0213 19:12:04.851643 55174 cli_runner.go:164] Run: docker container inspect old-k8s-version-187000 --format={{.State.Status}}
I0213 19:12:04.909545 55174 cli_runner.go:164] Run: docker exec old-k8s-version-187000 stat /var/lib/dpkg/alternatives/iptables
I0213 19:12:05.015130 55174 oci.go:144] the created container "old-k8s-version-187000" has a running status.
I0213 19:12:05.015197 55174 kic.go:225] Creating ssh key for kic: /Users/jenkins/minikube-integration/18165-38421/.minikube/machines/old-k8s-version-187000/id_rsa...
I0213 19:12:05.151974 55174 kic_runner.go:191] docker (temp): /Users/jenkins/minikube-integration/18165-38421/.minikube/machines/old-k8s-version-187000/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0213 19:12:05.247369 55174 cli_runner.go:164] Run: docker container inspect old-k8s-version-187000 --format={{.State.Status}}
I0213 19:12:05.305725 55174 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0213 19:12:05.305782 55174 kic_runner.go:114] Args: [docker exec --privileged old-k8s-version-187000 chown docker:docker /home/docker/.ssh/authorized_keys]
I0213 19:12:05.405766 55174 cli_runner.go:164] Run: docker container inspect old-k8s-version-187000 --format={{.State.Status}}
I0213 19:12:05.462884 55174 machine.go:88] provisioning docker machine ...
I0213 19:12:05.462934 55174 ubuntu.go:169] provisioning hostname "old-k8s-version-187000"
I0213 19:12:05.463036 55174 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-187000
I0213 19:12:05.517571 55174 main.go:141] libmachine: Using SSH client type: native
I0213 19:12:05.517897 55174 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1407080] 0x1409d60 <nil> [] 0s} 127.0.0.1 56980 <nil> <nil>}
I0213 19:12:05.517910 55174 main.go:141] libmachine: About to run SSH command:
sudo hostname old-k8s-version-187000 && echo "old-k8s-version-187000" | sudo tee /etc/hostname
I0213 19:12:05.682672 55174 main.go:141] libmachine: SSH cmd err, output: <nil>: old-k8s-version-187000
I0213 19:12:05.682809 55174 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-187000
I0213 19:12:05.735388 55174 main.go:141] libmachine: Using SSH client type: native
I0213 19:12:05.735683 55174 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1407080] 0x1409d60 <nil> [] 0s} 127.0.0.1 56980 <nil> <nil>}
I0213 19:12:05.735696 55174 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sold-k8s-version-187000' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 old-k8s-version-187000/g' /etc/hosts;
else
echo '127.0.1.1 old-k8s-version-187000' | sudo tee -a /etc/hosts;
fi
fi
I0213 19:12:05.873452 55174 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0213 19:12:05.873471 55174 ubuntu.go:175] set auth options {CertDir:/Users/jenkins/minikube-integration/18165-38421/.minikube CaCertPath:/Users/jenkins/minikube-integration/18165-38421/.minikube/certs/ca.pem CaPrivateKeyPath:/Users/jenkins/minikube-integration/18165-38421/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/Users/jenkins/minikube-integration/18165-38421/.minikube/machines/server.pem ServerKeyPath:/Users/jenkins/minikube-integration/18165-38421/.minikube/machines/server-key.pem ClientKeyPath:/Users/jenkins/minikube-integration/18165-38421/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/Users/jenkins/minikube-integration/18165-38421/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/Users/jenkins/minikube-integration/18165-38421/.minikube}
I0213 19:12:05.873488 55174 ubuntu.go:177] setting up certificates
I0213 19:12:05.873495 55174 provision.go:83] configureAuth start
I0213 19:12:05.873565 55174 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-187000
I0213 19:12:05.925098 55174 provision.go:138] copyHostCerts
I0213 19:12:05.925184 55174 exec_runner.go:144] found /Users/jenkins/minikube-integration/18165-38421/.minikube/ca.pem, removing ...
I0213 19:12:05.925193 55174 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/18165-38421/.minikube/ca.pem
I0213 19:12:05.925316 55174 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/18165-38421/.minikube/certs/ca.pem --> /Users/jenkins/minikube-integration/18165-38421/.minikube/ca.pem (1078 bytes)
I0213 19:12:05.925530 55174 exec_runner.go:144] found /Users/jenkins/minikube-integration/18165-38421/.minikube/cert.pem, removing ...
I0213 19:12:05.925537 55174 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/18165-38421/.minikube/cert.pem
I0213 19:12:05.925615 55174 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/18165-38421/.minikube/certs/cert.pem --> /Users/jenkins/minikube-integration/18165-38421/.minikube/cert.pem (1123 bytes)
I0213 19:12:05.925803 55174 exec_runner.go:144] found /Users/jenkins/minikube-integration/18165-38421/.minikube/key.pem, removing ...
I0213 19:12:05.925812 55174 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/18165-38421/.minikube/key.pem
I0213 19:12:05.925885 55174 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/18165-38421/.minikube/certs/key.pem --> /Users/jenkins/minikube-integration/18165-38421/.minikube/key.pem (1679 bytes)
I0213 19:12:05.926033 55174 provision.go:112] generating server cert: /Users/jenkins/minikube-integration/18165-38421/.minikube/machines/server.pem ca-key=/Users/jenkins/minikube-integration/18165-38421/.minikube/certs/ca.pem private-key=/Users/jenkins/minikube-integration/18165-38421/.minikube/certs/ca-key.pem org=jenkins.old-k8s-version-187000 san=[192.168.85.2 127.0.0.1 localhost 127.0.0.1 minikube old-k8s-version-187000]
I0213 19:12:06.027536 55174 provision.go:172] copyRemoteCerts
I0213 19:12:06.027740 55174 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0213 19:12:06.027796 55174 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-187000
I0213 19:12:06.080361 55174 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:56980 SSHKeyPath:/Users/jenkins/minikube-integration/18165-38421/.minikube/machines/old-k8s-version-187000/id_rsa Username:docker}
I0213 19:12:06.187382 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1078 bytes)
I0213 19:12:06.236736 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/machines/server.pem --> /etc/docker/server.pem (1241 bytes)
I0213 19:12:06.277231 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1675 bytes)
I0213 19:12:06.318807 55174 provision.go:86] duration metric: configureAuth took 445.282721ms
I0213 19:12:06.318821 55174 ubuntu.go:193] setting minikube options for container-runtime
I0213 19:12:06.319006 55174 config.go:182] Loaded profile config "old-k8s-version-187000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.16.0
I0213 19:12:06.319081 55174 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-187000
I0213 19:12:06.372540 55174 main.go:141] libmachine: Using SSH client type: native
I0213 19:12:06.372873 55174 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1407080] 0x1409d60 <nil> [] 0s} 127.0.0.1 56980 <nil> <nil>}
I0213 19:12:06.372888 55174 main.go:141] libmachine: About to run SSH command:
df --output=fstype / | tail -n 1
I0213 19:12:06.509951 55174 main.go:141] libmachine: SSH cmd err, output: <nil>: overlay
I0213 19:12:06.509968 55174 ubuntu.go:71] root file system type: overlay
I0213 19:12:06.510061 55174 provision.go:309] Updating docker unit: /lib/systemd/system/docker.service ...
I0213 19:12:06.510153 55174 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-187000
I0213 19:12:06.561223 55174 main.go:141] libmachine: Using SSH client type: native
I0213 19:12:06.561537 55174 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1407080] 0x1409d60 <nil> [] 0s} 127.0.0.1 56980 <nil> <nil>}
I0213 19:12:06.561586 55174 main.go:141] libmachine: About to run SSH command:
sudo mkdir -p /lib/systemd/system && printf %s "[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP \$MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service.new
I0213 19:12:06.722471 55174 main.go:141] libmachine: SSH cmd err, output: <nil>: [Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
I0213 19:12:06.722568 55174 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-187000
I0213 19:12:06.775752 55174 main.go:141] libmachine: Using SSH client type: native
I0213 19:12:06.776060 55174 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1407080] 0x1409d60 <nil> [] 0s} 127.0.0.1 56980 <nil> <nil>}
I0213 19:12:06.776074 55174 main.go:141] libmachine: About to run SSH command:
sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
I0213 19:12:07.419890 55174 main.go:141] libmachine: SSH cmd err, output: <nil>: --- /lib/systemd/system/docker.service 2023-10-26 09:06:22.000000000 +0000
+++ /lib/systemd/system/docker.service.new 2024-02-14 03:12:06.717243524 +0000
@@ -1,30 +1,32 @@
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
-After=network-online.target docker.socket firewalld.service containerd.service time-set.target
-Wants=network-online.target containerd.service
+BindsTo=containerd.service
+After=network-online.target firewalld.service containerd.service
+Wants=network-online.target
Requires=docker.socket
+StartLimitBurst=3
+StartLimitIntervalSec=60
[Service]
Type=notify
-# the default is not to use systemd for cgroups because the delegate issues still
-# exists and systemd currently does not support the cgroup feature set required
-# for containers run by docker
-ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
-ExecReload=/bin/kill -s HUP $MAINPID
-TimeoutStartSec=0
-RestartSec=2
-Restart=always
+Restart=on-failure
-# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
-# Both the old, and new location are accepted by systemd 229 and up, so using the old location
-# to make them work for either version of systemd.
-StartLimitBurst=3
-# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
-# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
-# this option work for either version of systemd.
-StartLimitInterval=60s
+
+# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
+# The base configuration already specifies an 'ExecStart=...' command. The first directive
+# here is to clear out that command inherited from the base configuration. Without this,
+# the command from the base configuration and the command specified here are treated as
+# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
+# will catch this invalid input and refuse to start the service with an error like:
+# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
+
+# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
+# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
+ExecStart=
+ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
+ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
@@ -32,16 +34,16 @@
LimitNPROC=infinity
LimitCORE=infinity
-# Comment TasksMax if your systemd version does not support it.
-# Only systemd 226 and above support this option.
+# Uncomment TasksMax if your systemd version supports it.
+# Only systemd 226 and above support this version.
TasksMax=infinity
+TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
-OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
I0213 19:12:07.419920 55174 machine.go:91] provisioned docker machine in 1.957024412s
I0213 19:12:07.419928 55174 client.go:171] LocalClient.Create took 5.881638013s
I0213 19:12:07.419944 55174 start.go:167] duration metric: libmachine.API.Create for "old-k8s-version-187000" took 5.88168893s
I0213 19:12:07.419951 55174 start.go:300] post-start starting for "old-k8s-version-187000" (driver="docker")
I0213 19:12:07.419958 55174 start.go:329] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0213 19:12:07.420036 55174 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0213 19:12:07.420093 55174 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-187000
I0213 19:12:07.473644 55174 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:56980 SSHKeyPath:/Users/jenkins/minikube-integration/18165-38421/.minikube/machines/old-k8s-version-187000/id_rsa Username:docker}
I0213 19:12:07.580277 55174 ssh_runner.go:195] Run: cat /etc/os-release
I0213 19:12:07.584381 55174 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0213 19:12:07.584406 55174 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0213 19:12:07.584414 55174 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0213 19:12:07.584419 55174 info.go:137] Remote host: Ubuntu 22.04.3 LTS
I0213 19:12:07.584429 55174 filesync.go:126] Scanning /Users/jenkins/minikube-integration/18165-38421/.minikube/addons for local assets ...
I0213 19:12:07.584524 55174 filesync.go:126] Scanning /Users/jenkins/minikube-integration/18165-38421/.minikube/files for local assets ...
I0213 19:12:07.584735 55174 filesync.go:149] local asset: /Users/jenkins/minikube-integration/18165-38421/.minikube/files/etc/ssl/certs/388992.pem -> 388992.pem in /etc/ssl/certs
I0213 19:12:07.584937 55174 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0213 19:12:07.599533 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/files/etc/ssl/certs/388992.pem --> /etc/ssl/certs/388992.pem (1708 bytes)
I0213 19:12:07.640919 55174 start.go:303] post-start completed in 220.960743ms
I0213 19:12:07.641460 55174 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-187000
I0213 19:12:07.693767 55174 profile.go:148] Saving config to /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/config.json ...
I0213 19:12:07.694250 55174 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0213 19:12:07.694316 55174 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-187000
I0213 19:12:07.746263 55174 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:56980 SSHKeyPath:/Users/jenkins/minikube-integration/18165-38421/.minikube/machines/old-k8s-version-187000/id_rsa Username:docker}
I0213 19:12:07.838254 55174 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0213 19:12:07.843321 55174 start.go:128] duration metric: createHost completed in 6.32663669s
I0213 19:12:07.843344 55174 start.go:83] releasing machines lock for "old-k8s-version-187000", held for 6.326774128s
I0213 19:12:07.843438 55174 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-187000
I0213 19:12:07.894941 55174 ssh_runner.go:195] Run: cat /version.json
I0213 19:12:07.894965 55174 ssh_runner.go:195] Run: curl -sS -m 2 https://registry.k8s.io/
I0213 19:12:07.895019 55174 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-187000
I0213 19:12:07.895049 55174 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-187000
I0213 19:12:07.953673 55174 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:56980 SSHKeyPath:/Users/jenkins/minikube-integration/18165-38421/.minikube/machines/old-k8s-version-187000/id_rsa Username:docker}
I0213 19:12:07.953660 55174 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:56980 SSHKeyPath:/Users/jenkins/minikube-integration/18165-38421/.minikube/machines/old-k8s-version-187000/id_rsa Username:docker}
I0213 19:12:08.151285 55174 ssh_runner.go:195] Run: systemctl --version
I0213 19:12:08.156153 55174 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
I0213 19:12:08.161333 55174 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *loopback.conf* -not -name *.mk_disabled -exec sh -c "grep -q loopback {} && ( grep -q name {} || sudo sed -i '/"type": "loopback"/i \ \ \ \ "name": "loopback",' {} ) && sudo sed -i 's|"cniVersion": ".*"|"cniVersion": "1.0.0"|g' {}" ;
I0213 19:12:08.203293 55174 cni.go:230] loopback cni configuration patched: "/etc/cni/net.d/*loopback.conf*" found
I0213 19:12:08.203452 55174 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *bridge* -not -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e '/"dst": ".*:.*"/d' -e 's|^(.*)"dst": (.*)[,*]$|\1"dst": \2|g' -e '/"subnet": ".*:.*"/d' -e 's|^(.*)"subnet": ".*"(.*)[,*]$|\1"subnet": "10.244.0.0/16"\2|g' {}" ;
I0213 19:12:08.231676 55174 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e 's|^(.*)"subnet": ".*"(.*)$|\1"subnet": "10.244.0.0/16"\2|g' -e 's|^(.*)"gateway": ".*"(.*)$|\1"gateway": "10.244.0.1"\2|g' {}" ;
I0213 19:12:08.259309 55174 cni.go:308] configured [/etc/cni/net.d/100-crio-bridge.conf, /etc/cni/net.d/87-podman-bridge.conflist] bridge cni config(s)
I0213 19:12:08.259325 55174 start.go:475] detecting cgroup driver to use...
I0213 19:12:08.259337 55174 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0213 19:12:08.259426 55174 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0213 19:12:08.287855 55174 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.1"|' /etc/containerd/config.toml"
I0213 19:12:08.303842 55174 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I0213 19:12:08.320136 55174 containerd.go:146] configuring containerd to use "cgroupfs" as cgroup driver...
I0213 19:12:08.320202 55174 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = false|g' /etc/containerd/config.toml"
I0213 19:12:08.337972 55174 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0213 19:12:08.354122 55174 ssh_runner.go:195] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I0213 19:12:08.371002 55174 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0213 19:12:08.388519 55174 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0213 19:12:08.404717 55174 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I0213 19:12:08.421599 55174 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0213 19:12:08.438025 55174 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0213 19:12:08.453367 55174 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0213 19:12:08.523203 55174 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0213 19:12:08.619050 55174 start.go:475] detecting cgroup driver to use...
I0213 19:12:08.619095 55174 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0213 19:12:08.619184 55174 ssh_runner.go:195] Run: sudo systemctl cat docker.service
I0213 19:12:08.649190 55174 cruntime.go:279] skipping containerd shutdown because we are bound to it
I0213 19:12:08.649276 55174 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0213 19:12:08.678020 55174 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///var/run/dockershim.sock
" | sudo tee /etc/crictl.yaml"
I0213 19:12:08.715623 55174 ssh_runner.go:195] Run: which cri-dockerd
I0213 19:12:08.721995 55174 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/cri-docker.service.d
I0213 19:12:08.745816 55174 ssh_runner.go:362] scp memory --> /etc/systemd/system/cri-docker.service.d/10-cni.conf (189 bytes)
I0213 19:12:08.788206 55174 ssh_runner.go:195] Run: sudo systemctl unmask docker.service
I0213 19:12:08.905489 55174 ssh_runner.go:195] Run: sudo systemctl enable docker.socket
I0213 19:12:09.017318 55174 docker.go:574] configuring docker to use "cgroupfs" as cgroup driver...
I0213 19:12:09.017408 55174 ssh_runner.go:362] scp memory --> /etc/docker/daemon.json (130 bytes)
I0213 19:12:09.053113 55174 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0213 19:12:09.124855 55174 ssh_runner.go:195] Run: sudo systemctl restart docker
I0213 19:12:09.453034 55174 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0213 19:12:09.481568 55174 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0213 19:12:09.559690 55174 out.go:204] * Preparing Kubernetes v1.16.0 on Docker 24.0.7 ...
I0213 19:12:09.559798 55174 cli_runner.go:164] Run: docker exec -t old-k8s-version-187000 dig +short host.docker.internal
I0213 19:12:09.693621 55174 network.go:96] got host ip for mount in container by digging dns: 192.168.65.254
I0213 19:12:09.693851 55174 ssh_runner.go:195] Run: grep 192.168.65.254 host.minikube.internal$ /etc/hosts
I0213 19:12:09.699649 55174 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.65.254 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0213 19:12:09.718873 55174 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "8443/tcp") 0).HostPort}}'" old-k8s-version-187000
I0213 19:12:09.778212 55174 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0213 19:12:09.778296 55174 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0213 19:12:09.799554 55174 docker.go:685] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0213 19:12:09.799570 55174 docker.go:691] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0213 19:12:09.799637 55174 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0213 19:12:09.815885 55174 ssh_runner.go:195] Run: which lz4
I0213 19:12:09.821035 55174 ssh_runner.go:195] Run: stat -c "%s %y" /preloaded.tar.lz4
I0213 19:12:09.825821 55174 ssh_runner.go:352] existence check for /preloaded.tar.lz4: stat -c "%s %y" /preloaded.tar.lz4: Process exited with status 1
stdout:
stderr:
stat: cannot statx '/preloaded.tar.lz4': No such file or directory
I0213 19:12:09.825859 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 --> /preloaded.tar.lz4 (369789069 bytes)
I0213 19:12:16.110628 55174 docker.go:649] Took 6.289711 seconds to copy over tarball
I0213 19:12:16.110737 55174 ssh_runner.go:195] Run: sudo tar --xattrs --xattrs-include security.capability -I lz4 -C /var -xf /preloaded.tar.lz4
I0213 19:12:17.752414 55174 ssh_runner.go:235] Completed: sudo tar --xattrs --xattrs-include security.capability -I lz4 -C /var -xf /preloaded.tar.lz4: (1.641671892s)
I0213 19:12:17.752430 55174 ssh_runner.go:146] rm: /preloaded.tar.lz4
I0213 19:12:17.805984 55174 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0213 19:12:17.822950 55174 ssh_runner.go:362] scp memory --> /var/lib/docker/image/overlay2/repositories.json (2499 bytes)
I0213 19:12:17.856224 55174 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0213 19:12:17.926279 55174 ssh_runner.go:195] Run: sudo systemctl restart docker
I0213 19:12:18.426382 55174 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0213 19:12:18.448246 55174 docker.go:685] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0213 19:12:18.448258 55174 docker.go:691] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0213 19:12:18.448267 55174 cache_images.go:88] LoadImages start: [registry.k8s.io/kube-apiserver:v1.16.0 registry.k8s.io/kube-controller-manager:v1.16.0 registry.k8s.io/kube-scheduler:v1.16.0 registry.k8s.io/kube-proxy:v1.16.0 registry.k8s.io/pause:3.1 registry.k8s.io/etcd:3.3.15-0 registry.k8s.io/coredns:1.6.2 gcr.io/k8s-minikube/storage-provisioner:v5]
I0213 19:12:18.453262 55174 image.go:134] retrieving image: registry.k8s.io/kube-proxy:v1.16.0
I0213 19:12:18.453356 55174 image.go:134] retrieving image: registry.k8s.io/etcd:3.3.15-0
I0213 19:12:18.453477 55174 image.go:134] retrieving image: registry.k8s.io/pause:3.1
I0213 19:12:18.453513 55174 image.go:134] retrieving image: registry.k8s.io/kube-scheduler:v1.16.0
I0213 19:12:18.453504 55174 image.go:134] retrieving image: registry.k8s.io/kube-controller-manager:v1.16.0
I0213 19:12:18.454091 55174 image.go:134] retrieving image: registry.k8s.io/coredns:1.6.2
I0213 19:12:18.454515 55174 image.go:134] retrieving image: registry.k8s.io/kube-apiserver:v1.16.0
I0213 19:12:18.454603 55174 image.go:134] retrieving image: gcr.io/k8s-minikube/storage-provisioner:v5
I0213 19:12:18.459982 55174 image.go:177] daemon lookup for registry.k8s.io/pause:3.1: Error response from daemon: No such image: registry.k8s.io/pause:3.1
I0213 19:12:18.460101 55174 image.go:177] daemon lookup for registry.k8s.io/kube-scheduler:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-scheduler:v1.16.0
I0213 19:12:18.460138 55174 image.go:177] daemon lookup for registry.k8s.io/kube-controller-manager:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-controller-manager:v1.16.0
I0213 19:12:18.460247 55174 image.go:177] daemon lookup for registry.k8s.io/coredns:1.6.2: Error response from daemon: No such image: registry.k8s.io/coredns:1.6.2
I0213 19:12:18.461349 55174 image.go:177] daemon lookup for registry.k8s.io/kube-apiserver:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-apiserver:v1.16.0
I0213 19:12:18.461435 55174 image.go:177] daemon lookup for registry.k8s.io/etcd:3.3.15-0: Error response from daemon: No such image: registry.k8s.io/etcd:3.3.15-0
I0213 19:12:18.461560 55174 image.go:177] daemon lookup for registry.k8s.io/kube-proxy:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-proxy:v1.16.0
I0213 19:12:18.461453 55174 image.go:177] daemon lookup for gcr.io/k8s-minikube/storage-provisioner:v5: Error response from daemon: No such image: gcr.io/k8s-minikube/storage-provisioner:v5
I0213 19:12:20.339272 55174 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-apiserver:v1.16.0
I0213 19:12:20.358916 55174 cache_images.go:116] "registry.k8s.io/kube-apiserver:v1.16.0" needs transfer: "registry.k8s.io/kube-apiserver:v1.16.0" does not exist at hash "b305571ca60a5a7818bda47da122683d75e8a1907475681ee8b1efbd06bff12e" in container runtime
I0213 19:12:20.358957 55174 docker.go:337] Removing image: registry.k8s.io/kube-apiserver:v1.16.0
I0213 19:12:20.359038 55174 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-apiserver:v1.16.0
I0213 19:12:20.377553 55174 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/18165-38421/.minikube/cache/images/amd64/registry.k8s.io/kube-apiserver_v1.16.0
I0213 19:12:20.431127 55174 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-scheduler:v1.16.0
I0213 19:12:20.451413 55174 cache_images.go:116] "registry.k8s.io/kube-scheduler:v1.16.0" needs transfer: "registry.k8s.io/kube-scheduler:v1.16.0" does not exist at hash "301ddc62b80b16315d3c2653cf3888370394277afb3187614cfa20edc352ca0a" in container runtime
I0213 19:12:20.451437 55174 docker.go:337] Removing image: registry.k8s.io/kube-scheduler:v1.16.0
I0213 19:12:20.451491 55174 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-scheduler:v1.16.0
I0213 19:12:20.471349 55174 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/18165-38421/.minikube/cache/images/amd64/registry.k8s.io/kube-scheduler_v1.16.0
I0213 19:12:20.478964 55174 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-proxy:v1.16.0
I0213 19:12:20.479679 55174 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-controller-manager:v1.16.0
I0213 19:12:20.490202 55174 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/pause:3.1
I0213 19:12:20.499922 55174 cache_images.go:116] "registry.k8s.io/kube-proxy:v1.16.0" needs transfer: "registry.k8s.io/kube-proxy:v1.16.0" does not exist at hash "c21b0c7400f988db4777858edd13b6d3930d62d7ccf026d2415485a52037f384" in container runtime
I0213 19:12:20.499959 55174 docker.go:337] Removing image: registry.k8s.io/kube-proxy:v1.16.0
I0213 19:12:20.500023 55174 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-proxy:v1.16.0
I0213 19:12:20.501059 55174 cache_images.go:116] "registry.k8s.io/kube-controller-manager:v1.16.0" needs transfer: "registry.k8s.io/kube-controller-manager:v1.16.0" does not exist at hash "06a629a7e51cdcc81a5ed6a3e6650348312f20c954ac52ee489a023628ec9c7d" in container runtime
I0213 19:12:20.501080 55174 docker.go:337] Removing image: registry.k8s.io/kube-controller-manager:v1.16.0
I0213 19:12:20.501142 55174 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-controller-manager:v1.16.0
I0213 19:12:20.512182 55174 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/coredns:1.6.2
I0213 19:12:20.512822 55174 cache_images.go:116] "registry.k8s.io/pause:3.1" needs transfer: "registry.k8s.io/pause:3.1" does not exist at hash "da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e" in container runtime
I0213 19:12:20.512849 55174 docker.go:337] Removing image: registry.k8s.io/pause:3.1
I0213 19:12:20.512939 55174 ssh_runner.go:195] Run: docker rmi registry.k8s.io/pause:3.1
I0213 19:12:20.523430 55174 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/etcd:3.3.15-0
I0213 19:12:20.525195 55174 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/18165-38421/.minikube/cache/images/amd64/registry.k8s.io/kube-proxy_v1.16.0
I0213 19:12:20.525764 55174 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/18165-38421/.minikube/cache/images/amd64/registry.k8s.io/kube-controller-manager_v1.16.0
I0213 19:12:20.538437 55174 cache_images.go:116] "registry.k8s.io/coredns:1.6.2" needs transfer: "registry.k8s.io/coredns:1.6.2" does not exist at hash "bf261d157914477ee1a5969d28ec687f3fbfc9fb5a664b22df78e57023b0e03b" in container runtime
I0213 19:12:20.538468 55174 docker.go:337] Removing image: registry.k8s.io/coredns:1.6.2
I0213 19:12:20.538547 55174 ssh_runner.go:195] Run: docker rmi registry.k8s.io/coredns:1.6.2
I0213 19:12:20.541335 55174 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/18165-38421/.minikube/cache/images/amd64/registry.k8s.io/pause_3.1
I0213 19:12:20.594256 55174 cache_images.go:116] "registry.k8s.io/etcd:3.3.15-0" needs transfer: "registry.k8s.io/etcd:3.3.15-0" does not exist at hash "b2756210eeabf84f3221da9959e9483f3919dc2aaab4cd45e7cd072fcbde27ed" in container runtime
I0213 19:12:20.594282 55174 docker.go:337] Removing image: registry.k8s.io/etcd:3.3.15-0
I0213 19:12:20.594343 55174 ssh_runner.go:195] Run: docker rmi registry.k8s.io/etcd:3.3.15-0
I0213 19:12:20.605733 55174 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/18165-38421/.minikube/cache/images/amd64/registry.k8s.io/coredns_1.6.2
I0213 19:12:20.615016 55174 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/18165-38421/.minikube/cache/images/amd64/registry.k8s.io/etcd_3.3.15-0
I0213 19:12:21.242662 55174 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} gcr.io/k8s-minikube/storage-provisioner:v5
I0213 19:12:21.263161 55174 cache_images.go:92] LoadImages completed in 2.814897668s
W0213 19:12:21.263217 55174 out.go:239] X Unable to load cached images: loading cached images: stat /Users/jenkins/minikube-integration/18165-38421/.minikube/cache/images/amd64/registry.k8s.io/kube-apiserver_v1.16.0: no such file or directory
X Unable to load cached images: loading cached images: stat /Users/jenkins/minikube-integration/18165-38421/.minikube/cache/images/amd64/registry.k8s.io/kube-apiserver_v1.16.0: no such file or directory
I0213 19:12:21.263296 55174 ssh_runner.go:195] Run: docker info --format {{.CgroupDriver}}
I0213 19:12:21.314323 55174 cni.go:84] Creating CNI manager for ""
I0213 19:12:21.314340 55174 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0213 19:12:21.314355 55174 kubeadm.go:87] Using pod CIDR: 10.244.0.0/16
I0213 19:12:21.314371 55174 kubeadm.go:176] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.85.2 APIServerPort:8443 KubernetesVersion:v1.16.0 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:old-k8s-version-187000 NodeName:old-k8s-version-187000 DNSDomain:cluster.local CRISocket:/var/run/dockershim.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.85.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.85.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticP
odPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[hairpinMode:hairpin-veth runtimeRequestTimeout:15m] PrependCriSocketUnix:false}
I0213 19:12:21.314459 55174 kubeadm.go:181] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.85.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: "old-k8s-version-187000"
kubeletExtraArgs:
node-ip: 192.168.85.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.85.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: old-k8s-version-187000
controlPlaneEndpoint: control-plane.minikube.internal:8443
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
listen-metrics-urls: http://127.0.0.1:2381,http://192.168.85.2:2381
kubernetesVersion: v1.16.0
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0213 19:12:21.314506 55174 kubeadm.go:976] kubelet [Unit]
Wants=docker.socket
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.16.0/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=old-k8s-version-187000 --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.85.2
[Install]
config:
{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-187000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:}
I0213 19:12:21.314566 55174 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.16.0
I0213 19:12:21.329231 55174 binaries.go:44] Found k8s binaries, skipping transfer
I0213 19:12:21.329299 55174 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0213 19:12:21.344368 55174 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (348 bytes)
I0213 19:12:21.373022 55174 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0213 19:12:21.402054 55174 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2174 bytes)
I0213 19:12:21.430642 55174 ssh_runner.go:195] Run: grep 192.168.85.2 control-plane.minikube.internal$ /etc/hosts
I0213 19:12:21.435332 55174 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.85.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0213 19:12:21.452532 55174 certs.go:56] Setting up /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000 for IP: 192.168.85.2
I0213 19:12:21.452553 55174 certs.go:190] acquiring lock for shared ca certs: {Name:mkc5f1a81e3b2f96d4314e8cdee92a3e3396cb89 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 19:12:21.452726 55174 certs.go:199] skipping minikubeCA CA generation: /Users/jenkins/minikube-integration/18165-38421/.minikube/ca.key
I0213 19:12:21.452796 55174 certs.go:199] skipping proxyClientCA CA generation: /Users/jenkins/minikube-integration/18165-38421/.minikube/proxy-client-ca.key
I0213 19:12:21.452840 55174 certs.go:319] generating minikube-user signed cert: /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/client.key
I0213 19:12:21.452856 55174 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/client.crt with IP's: []
I0213 19:12:21.580497 55174 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/client.crt ...
I0213 19:12:21.580512 55174 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/client.crt: {Name:mk2b122c169061bb87f56a32f0b0ff9c71c53588 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 19:12:21.580860 55174 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/client.key ...
I0213 19:12:21.580869 55174 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/client.key: {Name:mka968d5d957b95e8bd77e593629ef65cd01a944 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 19:12:21.581078 55174 certs.go:319] generating minikube signed cert: /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/apiserver.key.43b9df8c
I0213 19:12:21.581093 55174 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/apiserver.crt.43b9df8c with IP's: [192.168.85.2 10.96.0.1 127.0.0.1 10.0.0.1]
I0213 19:12:21.714726 55174 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/apiserver.crt.43b9df8c ...
I0213 19:12:21.714741 55174 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/apiserver.crt.43b9df8c: {Name:mkc23ac9979a6ed24c0b56a4b50e79083da52573 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 19:12:21.715097 55174 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/apiserver.key.43b9df8c ...
I0213 19:12:21.715110 55174 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/apiserver.key.43b9df8c: {Name:mke958fbf3294f0f81df20c78998534e19113910 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 19:12:21.715355 55174 certs.go:337] copying /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/apiserver.crt.43b9df8c -> /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/apiserver.crt
I0213 19:12:21.715557 55174 certs.go:341] copying /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/apiserver.key.43b9df8c -> /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/apiserver.key
I0213 19:12:21.715811 55174 certs.go:319] generating aggregator signed cert: /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/proxy-client.key
I0213 19:12:21.715826 55174 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/proxy-client.crt with IP's: []
I0213 19:12:21.799235 55174 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/proxy-client.crt ...
I0213 19:12:21.799247 55174 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/proxy-client.crt: {Name:mk398667ce44e6e73fa533363b9b5d5315045450 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 19:12:21.799519 55174 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/proxy-client.key ...
I0213 19:12:21.799528 55174 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/proxy-client.key: {Name:mk25a2afe532ca387a7a1ea9ab2a1da594c59ab8 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0213 19:12:21.799916 55174 certs.go:437] found cert: /Users/jenkins/minikube-integration/18165-38421/.minikube/certs/Users/jenkins/minikube-integration/18165-38421/.minikube/certs/38899.pem (1338 bytes)
W0213 19:12:21.800004 55174 certs.go:433] ignoring /Users/jenkins/minikube-integration/18165-38421/.minikube/certs/Users/jenkins/minikube-integration/18165-38421/.minikube/certs/38899_empty.pem, impossibly tiny 0 bytes
I0213 19:12:21.800016 55174 certs.go:437] found cert: /Users/jenkins/minikube-integration/18165-38421/.minikube/certs/Users/jenkins/minikube-integration/18165-38421/.minikube/certs/ca-key.pem (1675 bytes)
I0213 19:12:21.800116 55174 certs.go:437] found cert: /Users/jenkins/minikube-integration/18165-38421/.minikube/certs/Users/jenkins/minikube-integration/18165-38421/.minikube/certs/ca.pem (1078 bytes)
I0213 19:12:21.800150 55174 certs.go:437] found cert: /Users/jenkins/minikube-integration/18165-38421/.minikube/certs/Users/jenkins/minikube-integration/18165-38421/.minikube/certs/cert.pem (1123 bytes)
I0213 19:12:21.800181 55174 certs.go:437] found cert: /Users/jenkins/minikube-integration/18165-38421/.minikube/certs/Users/jenkins/minikube-integration/18165-38421/.minikube/certs/key.pem (1679 bytes)
I0213 19:12:21.800248 55174 certs.go:437] found cert: /Users/jenkins/minikube-integration/18165-38421/.minikube/files/etc/ssl/certs/Users/jenkins/minikube-integration/18165-38421/.minikube/files/etc/ssl/certs/388992.pem (1708 bytes)
I0213 19:12:21.800745 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1399 bytes)
I0213 19:12:21.841679 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1675 bytes)
I0213 19:12:21.881219 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0213 19:12:21.921499 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/old-k8s-version-187000/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1679 bytes)
I0213 19:12:21.961411 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0213 19:12:22.002016 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1679 bytes)
I0213 19:12:22.042226 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0213 19:12:22.082889 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1679 bytes)
I0213 19:12:22.123110 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0213 19:12:22.163461 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/certs/38899.pem --> /usr/share/ca-certificates/38899.pem (1338 bytes)
I0213 19:12:22.203551 55174 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/18165-38421/.minikube/files/etc/ssl/certs/388992.pem --> /usr/share/ca-certificates/388992.pem (1708 bytes)
I0213 19:12:22.244256 55174 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0213 19:12:22.272917 55174 ssh_runner.go:195] Run: openssl version
I0213 19:12:22.278557 55174 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0213 19:12:22.294380 55174 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0213 19:12:22.298538 55174 certs.go:480] hashing: -rw-r--r-- 1 root root 1111 Feb 14 02:09 /usr/share/ca-certificates/minikubeCA.pem
I0213 19:12:22.298593 55174 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0213 19:12:22.305246 55174 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0213 19:12:22.321687 55174 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/38899.pem && ln -fs /usr/share/ca-certificates/38899.pem /etc/ssl/certs/38899.pem"
I0213 19:12:22.337104 55174 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/38899.pem
I0213 19:12:22.341609 55174 certs.go:480] hashing: -rw-r--r-- 1 root root 1338 Feb 14 02:17 /usr/share/ca-certificates/38899.pem
I0213 19:12:22.341674 55174 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/38899.pem
I0213 19:12:22.348316 55174 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/38899.pem /etc/ssl/certs/51391683.0"
I0213 19:12:22.364240 55174 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/388992.pem && ln -fs /usr/share/ca-certificates/388992.pem /etc/ssl/certs/388992.pem"
I0213 19:12:22.379686 55174 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/388992.pem
I0213 19:12:22.384050 55174 certs.go:480] hashing: -rw-r--r-- 1 root root 1708 Feb 14 02:17 /usr/share/ca-certificates/388992.pem
I0213 19:12:22.384100 55174 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/388992.pem
I0213 19:12:22.390846 55174 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/388992.pem /etc/ssl/certs/3ec20f2e.0"
I0213 19:12:22.407180 55174 ssh_runner.go:195] Run: ls /var/lib/minikube/certs/etcd
I0213 19:12:22.411412 55174 certs.go:353] certs directory doesn't exist, likely first start: ls /var/lib/minikube/certs/etcd: Process exited with status 2
stdout:
stderr:
ls: cannot access '/var/lib/minikube/certs/etcd': No such file or directory
I0213 19:12:22.411459 55174 kubeadm.go:404] StartCluster: {Name:old-k8s-version-187000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-187000 Namespace:default APIServerName:minikubeCA APISer
verNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.85.2 Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false Disab
leMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs:}
I0213 19:12:22.411550 55174 ssh_runner.go:195] Run: docker ps --filter status=paused --filter=name=k8s_.*_(kube-system)_ --format={{.ID}}
I0213 19:12:22.430897 55174 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0213 19:12:22.445701 55174 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0213 19:12:22.460564 55174 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0213 19:12:22.460630 55174 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0213 19:12:22.475504 55174 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0213 19:12:22.475537 55174 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0213 19:12:22.532015 55174 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0213 19:12:22.532088 55174 kubeadm.go:322] [preflight] Running pre-flight checks
I0213 19:12:22.823495 55174 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0213 19:12:22.823582 55174 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0213 19:12:22.823655 55174 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0213 19:12:23.006287 55174 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0213 19:12:23.007071 55174 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0213 19:12:23.013227 55174 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0213 19:12:23.082286 55174 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0213 19:12:23.105244 55174 out.go:204] - Generating certificates and keys ...
I0213 19:12:23.105317 55174 kubeadm.go:322] [certs] Using existing ca certificate authority
I0213 19:12:23.105378 55174 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0213 19:12:23.199746 55174 kubeadm.go:322] [certs] Generating "apiserver-kubelet-client" certificate and key
I0213 19:12:23.263546 55174 kubeadm.go:322] [certs] Generating "front-proxy-ca" certificate and key
I0213 19:12:23.362671 55174 kubeadm.go:322] [certs] Generating "front-proxy-client" certificate and key
I0213 19:12:23.561641 55174 kubeadm.go:322] [certs] Generating "etcd/ca" certificate and key
I0213 19:12:23.681656 55174 kubeadm.go:322] [certs] Generating "etcd/server" certificate and key
I0213 19:12:23.682055 55174 kubeadm.go:322] [certs] etcd/server serving cert is signed for DNS names [old-k8s-version-187000 localhost] and IPs [192.168.85.2 127.0.0.1 ::1]
I0213 19:12:23.985729 55174 kubeadm.go:322] [certs] Generating "etcd/peer" certificate and key
I0213 19:12:23.986059 55174 kubeadm.go:322] [certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-187000 localhost] and IPs [192.168.85.2 127.0.0.1 ::1]
I0213 19:12:24.082387 55174 kubeadm.go:322] [certs] Generating "etcd/healthcheck-client" certificate and key
I0213 19:12:24.226428 55174 kubeadm.go:322] [certs] Generating "apiserver-etcd-client" certificate and key
I0213 19:12:24.489543 55174 kubeadm.go:322] [certs] Generating "sa" key and public key
I0213 19:12:24.489650 55174 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0213 19:12:24.663169 55174 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0213 19:12:24.702972 55174 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0213 19:12:24.806519 55174 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0213 19:12:25.190165 55174 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0213 19:12:25.190763 55174 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0213 19:12:25.213252 55174 out.go:204] - Booting up control plane ...
I0213 19:12:25.213364 55174 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0213 19:12:25.213459 55174 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0213 19:12:25.213547 55174 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0213 19:12:25.213648 55174 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0213 19:12:25.213863 55174 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0213 19:13:05.200808 55174 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0213 19:13:05.201131 55174 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 19:13:05.201270 55174 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 19:13:10.202893 55174 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 19:13:10.203050 55174 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 19:13:20.203994 55174 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 19:13:20.204155 55174 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 19:13:40.205737 55174 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 19:13:40.205892 55174 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 19:14:20.207289 55174 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 19:14:20.207473 55174 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 19:14:20.207484 55174 kubeadm.go:322]
I0213 19:14:20.207515 55174 kubeadm.go:322] Unfortunately, an error has occurred:
I0213 19:14:20.207558 55174 kubeadm.go:322] timed out waiting for the condition
I0213 19:14:20.207571 55174 kubeadm.go:322]
I0213 19:14:20.207607 55174 kubeadm.go:322] This error is likely caused by:
I0213 19:14:20.207645 55174 kubeadm.go:322] - The kubelet is not running
I0213 19:14:20.207734 55174 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0213 19:14:20.207744 55174 kubeadm.go:322]
I0213 19:14:20.207814 55174 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0213 19:14:20.207843 55174 kubeadm.go:322] - 'systemctl status kubelet'
I0213 19:14:20.207874 55174 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0213 19:14:20.207883 55174 kubeadm.go:322]
I0213 19:14:20.207967 55174 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0213 19:14:20.208042 55174 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0213 19:14:20.208103 55174 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0213 19:14:20.208135 55174 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0213 19:14:20.208185 55174 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0213 19:14:20.208207 55174 kubeadm.go:322] - 'docker logs CONTAINERID'
I0213 19:14:20.212464 55174 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0213 19:14:20.212558 55174 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0213 19:14:20.212673 55174 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
I0213 19:14:20.212766 55174 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0213 19:14:20.212827 55174 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0213 19:14:20.212879 55174 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
W0213 19:14:20.212962 55174 out.go:239] ! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-187000 localhost] and IPs [192.168.85.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-187000 localhost] and IPs [192.168.85.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-187000 localhost] and IPs [192.168.85.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-187000 localhost] and IPs [192.168.85.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
I0213 19:14:20.213006 55174 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm reset --cri-socket /var/run/dockershim.sock --force"
I0213 19:14:20.635774 55174 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service kubelet
I0213 19:14:20.652993 55174 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0213 19:14:20.653071 55174 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0213 19:14:20.668003 55174 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0213 19:14:20.668032 55174 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0213 19:14:20.723846 55174 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0213 19:14:20.723887 55174 kubeadm.go:322] [preflight] Running pre-flight checks
I0213 19:14:21.006613 55174 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0213 19:14:21.006698 55174 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0213 19:14:21.006778 55174 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0213 19:14:21.182490 55174 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0213 19:14:21.184535 55174 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0213 19:14:21.191438 55174 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0213 19:14:21.257769 55174 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0213 19:14:21.278823 55174 out.go:204] - Generating certificates and keys ...
I0213 19:14:21.278931 55174 kubeadm.go:322] [certs] Using existing ca certificate authority
I0213 19:14:21.279001 55174 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0213 19:14:21.279102 55174 kubeadm.go:322] [certs] Using existing apiserver-kubelet-client certificate and key on disk
I0213 19:14:21.279197 55174 kubeadm.go:322] [certs] Using existing front-proxy-ca certificate authority
I0213 19:14:21.279260 55174 kubeadm.go:322] [certs] Using existing front-proxy-client certificate and key on disk
I0213 19:14:21.279309 55174 kubeadm.go:322] [certs] Using existing etcd/ca certificate authority
I0213 19:14:21.279359 55174 kubeadm.go:322] [certs] Using existing etcd/server certificate and key on disk
I0213 19:14:21.279409 55174 kubeadm.go:322] [certs] Using existing etcd/peer certificate and key on disk
I0213 19:14:21.279461 55174 kubeadm.go:322] [certs] Using existing etcd/healthcheck-client certificate and key on disk
I0213 19:14:21.279524 55174 kubeadm.go:322] [certs] Using existing apiserver-etcd-client certificate and key on disk
I0213 19:14:21.279554 55174 kubeadm.go:322] [certs] Using the existing "sa" key
I0213 19:14:21.279600 55174 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0213 19:14:21.339870 55174 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0213 19:14:21.469940 55174 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0213 19:14:21.598676 55174 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0213 19:14:21.720525 55174 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0213 19:14:21.721223 55174 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0213 19:14:21.744068 55174 out.go:204] - Booting up control plane ...
I0213 19:14:21.744159 55174 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0213 19:14:21.744221 55174 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0213 19:14:21.744275 55174 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0213 19:14:21.744342 55174 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0213 19:14:21.744477 55174 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0213 19:15:01.730392 55174 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0213 19:15:01.731267 55174 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 19:15:01.731484 55174 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 19:15:06.732285 55174 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 19:15:06.732490 55174 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 19:15:16.734086 55174 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 19:15:16.734244 55174 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 19:15:36.735323 55174 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 19:15:36.735488 55174 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 19:16:16.737072 55174 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0213 19:16:16.737275 55174 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0213 19:16:16.737291 55174 kubeadm.go:322]
I0213 19:16:16.737351 55174 kubeadm.go:322] Unfortunately, an error has occurred:
I0213 19:16:16.737436 55174 kubeadm.go:322] timed out waiting for the condition
I0213 19:16:16.737453 55174 kubeadm.go:322]
I0213 19:16:16.737497 55174 kubeadm.go:322] This error is likely caused by:
I0213 19:16:16.737530 55174 kubeadm.go:322] - The kubelet is not running
I0213 19:16:16.737603 55174 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0213 19:16:16.737607 55174 kubeadm.go:322]
I0213 19:16:16.737680 55174 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0213 19:16:16.737708 55174 kubeadm.go:322] - 'systemctl status kubelet'
I0213 19:16:16.737731 55174 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0213 19:16:16.737760 55174 kubeadm.go:322]
I0213 19:16:16.737863 55174 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0213 19:16:16.737971 55174 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0213 19:16:16.738056 55174 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0213 19:16:16.738096 55174 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0213 19:16:16.738168 55174 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0213 19:16:16.738197 55174 kubeadm.go:322] - 'docker logs CONTAINERID'
I0213 19:16:16.742001 55174 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0213 19:16:16.742086 55174 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0213 19:16:16.742186 55174 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
I0213 19:16:16.742288 55174 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0213 19:16:16.742400 55174 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0213 19:16:16.742464 55174 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
I0213 19:16:16.742496 55174 kubeadm.go:406] StartCluster complete in 3m54.332612671s
I0213 19:16:16.742581 55174 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-apiserver --format={{.ID}}
I0213 19:16:16.760307 55174 logs.go:276] 0 containers: []
W0213 19:16:16.760321 55174 logs.go:278] No container was found matching "kube-apiserver"
I0213 19:16:16.760394 55174 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_etcd --format={{.ID}}
I0213 19:16:16.778333 55174 logs.go:276] 0 containers: []
W0213 19:16:16.778346 55174 logs.go:278] No container was found matching "etcd"
I0213 19:16:16.778416 55174 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_coredns --format={{.ID}}
I0213 19:16:16.795745 55174 logs.go:276] 0 containers: []
W0213 19:16:16.795759 55174 logs.go:278] No container was found matching "coredns"
I0213 19:16:16.795823 55174 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-scheduler --format={{.ID}}
I0213 19:16:16.815345 55174 logs.go:276] 0 containers: []
W0213 19:16:16.815359 55174 logs.go:278] No container was found matching "kube-scheduler"
I0213 19:16:16.815426 55174 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-proxy --format={{.ID}}
I0213 19:16:16.834729 55174 logs.go:276] 0 containers: []
W0213 19:16:16.834758 55174 logs.go:278] No container was found matching "kube-proxy"
I0213 19:16:16.834844 55174 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-controller-manager --format={{.ID}}
I0213 19:16:16.852782 55174 logs.go:276] 0 containers: []
W0213 19:16:16.852795 55174 logs.go:278] No container was found matching "kube-controller-manager"
I0213 19:16:16.852864 55174 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kindnet --format={{.ID}}
I0213 19:16:16.870039 55174 logs.go:276] 0 containers: []
W0213 19:16:16.870053 55174 logs.go:278] No container was found matching "kindnet"
I0213 19:16:16.870061 55174 logs.go:123] Gathering logs for container status ...
I0213 19:16:16.870068 55174 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0213 19:16:16.933906 55174 logs.go:123] Gathering logs for kubelet ...
I0213 19:16:16.933924 55174 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
I0213 19:16:16.982055 55174 logs.go:123] Gathering logs for dmesg ...
I0213 19:16:16.982074 55174 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0213 19:16:17.004424 55174 logs.go:123] Gathering logs for describe nodes ...
I0213 19:16:17.004443 55174 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0213 19:16:17.074652 55174 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0213 19:16:17.074663 55174 logs.go:123] Gathering logs for Docker ...
I0213 19:16:17.074672 55174 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u docker -u cri-docker -n 400"
W0213 19:16:17.097327 55174 out.go:364] Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0213 19:16:17.097349 55174 out.go:239] *
*
W0213 19:16:17.097384 55174 out.go:239] X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0213 19:16:17.097397 55174 out.go:239] *
*
W0213 19:16:17.098106 55174 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0213 19:16:17.161677 55174 out.go:177]
W0213 19:16:17.219680 55174 out.go:239] X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 24.0.7. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0213 19:16:17.219714 55174 out.go:239] * Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
* Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
W0213 19:16:17.219732 55174 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/4172
* Related issue: https://github.com/kubernetes/minikube/issues/4172
I0213 19:16:17.314604 55174 out.go:177]
** /stderr **
start_stop_delete_test.go:188: failed starting minikube -first start-. args "out/minikube-darwin-amd64 start -p old-k8s-version-187000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0": exit status 109
helpers_test.go:222: -----------------------post-mortem--------------------------------
helpers_test.go:230: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/FirstStart]: docker inspect <======
helpers_test.go:231: (dbg) Run: docker inspect old-k8s-version-187000
helpers_test.go:235: (dbg) docker inspect old-k8s-version-187000:
-- stdout --
[
{
"Id": "e0b9362b2efd09022b870091b98cdfc206a0380e70d1b74c3dc42ceb3e098e5f",
"Created": "2024-02-14T03:12:04.577549374Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 357312,
"ExitCode": 0,
"Error": "",
"StartedAt": "2024-02-14T03:12:04.787641413Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:9941de2e064a4a6a7155bfc66cedd2854b8c725b77bb8d4eaf81bef39f951dd7",
"ResolvConfPath": "/var/lib/docker/containers/e0b9362b2efd09022b870091b98cdfc206a0380e70d1b74c3dc42ceb3e098e5f/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/e0b9362b2efd09022b870091b98cdfc206a0380e70d1b74c3dc42ceb3e098e5f/hostname",
"HostsPath": "/var/lib/docker/containers/e0b9362b2efd09022b870091b98cdfc206a0380e70d1b74c3dc42ceb3e098e5f/hosts",
"LogPath": "/var/lib/docker/containers/e0b9362b2efd09022b870091b98cdfc206a0380e70d1b74c3dc42ceb3e098e5f/e0b9362b2efd09022b870091b98cdfc206a0380e70d1b74c3dc42ceb3e098e5f-json.log",
"Name": "/old-k8s-version-187000",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro",
"old-k8s-version-187000:/var"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "old-k8s-version-187000",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 2306867200,
"NanoCpus": 2000000000,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": [],
"BlkioDeviceWriteBps": [],
"BlkioDeviceReadIOps": [],
"BlkioDeviceWriteIOps": [],
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 2306867200,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": [],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/7c809f08c3fe15c84721952f204c528844488e74d4d3422d3f2c83b56532db72-init/diff:/var/lib/docker/overlay2/3ed0de4aac6b7e329f9acd865d0c22fc7cd3ad67bb85f95f8605165150fb68c8/diff",
"MergedDir": "/var/lib/docker/overlay2/7c809f08c3fe15c84721952f204c528844488e74d4d3422d3f2c83b56532db72/merged",
"UpperDir": "/var/lib/docker/overlay2/7c809f08c3fe15c84721952f204c528844488e74d4d3422d3f2c83b56532db72/diff",
"WorkDir": "/var/lib/docker/overlay2/7c809f08c3fe15c84721952f204c528844488e74d4d3422d3f2c83b56532db72/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "old-k8s-version-187000",
"Source": "/var/lib/docker/volumes/old-k8s-version-187000/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "old-k8s-version-187000",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"22/tcp": {},
"2376/tcp": {},
"32443/tcp": {},
"5000/tcp": {},
"8443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1704759386-17866@sha256:8c3c33047f9bc285e1f5f2a5aa14744a2fe04c58478f02f77b06169dea8dd3f0",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "old-k8s-version-187000",
"name.minikube.sigs.k8s.io": "old-k8s-version-187000",
"role.minikube.sigs.k8s.io": ""
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "27c338fc6291547d5f15f2c67ccf11fdad0b20a53d40597e882faa6e06914649",
"SandboxKey": "/var/run/docker/netns/27c338fc6291",
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "56980"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "56981"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "56982"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "56983"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "56979"
}
]
},
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"old-k8s-version-187000": {
"IPAMConfig": {
"IPv4Address": "192.168.85.2"
},
"Links": null,
"Aliases": [
"e0b9362b2efd",
"old-k8s-version-187000"
],
"MacAddress": "02:42:c0:a8:55:02",
"NetworkID": "4cb1b8693c9780c94ad8de0e0072aef11b304b625a6e68f12739c271830cb055",
"EndpointID": "fd354e3d40a2d2b72f7312573a95b4b0fb2ce3310b393493b5189e0c0efbbe51",
"Gateway": "192.168.85.1",
"IPAddress": "192.168.85.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"DriverOpts": null,
"DNSNames": [
"old-k8s-version-187000",
"e0b9362b2efd"
]
}
}
}
}
]
-- /stdout --
helpers_test.go:239: (dbg) Run: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-187000 -n old-k8s-version-187000
E0213 19:16:17.676483 38899 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/18165-38421/.minikube/profiles/calico-210000/client.crt: no such file or directory
helpers_test.go:239: (dbg) Non-zero exit: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-187000 -n old-k8s-version-187000: exit status 6 (406.569557ms)
-- stdout --
Running
WARNING: Your kubectl is pointing to stale minikube-vm.
To fix the kubectl context, run `minikube update-context`
-- /stdout --
** stderr **
E0213 19:16:17.878114 56040 status.go:415] kubeconfig endpoint: extract IP: "old-k8s-version-187000" does not appear in /Users/jenkins/minikube-integration/18165-38421/kubeconfig
** /stderr **
helpers_test.go:239: status error: exit status 6 (may be ok)
helpers_test.go:241: "old-k8s-version-187000" host is not running, skipping log retrieval (state="Running\nWARNING: Your kubectl is pointing to stale minikube-vm.\nTo fix the kubectl context, run `minikube update-context`")
--- FAIL: TestStartStop/group/old-k8s-version/serial/FirstStart (257.43s)