=== RUN TestStartStop/group/old-k8s-version/serial/FirstStart
start_stop_delete_test.go:186: (dbg) Run: out/minikube-darwin-amd64 start -p old-k8s-version-356000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0
E0216 09:45:55.269421 2151 cert_rotation.go:168] key failed with : open /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/calico-862000/client.crt: no such file or directory
start_stop_delete_test.go:186: (dbg) Non-zero exit: out/minikube-darwin-amd64 start -p old-k8s-version-356000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0: exit status 109 (4m16.921422685s)
-- stdout --
* [old-k8s-version-356000] minikube v1.32.0 on Darwin 14.3.1
- MINIKUBE_LOCATION=17936
- KUBECONFIG=/Users/jenkins/minikube-integration/17936-1021/kubeconfig
- MINIKUBE_BIN=out/minikube-darwin-amd64
- MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
- MINIKUBE_HOME=/Users/jenkins/minikube-integration/17936-1021/.minikube
- MINIKUBE_FORCE_SYSTEMD=
* Using the docker driver based on user configuration
* Using Docker Desktop driver with root privileges
* Starting control plane node old-k8s-version-356000 in cluster old-k8s-version-356000
* Pulling base image v0.0.42-1708008208-17936 ...
* Creating docker container (CPUs=2, Memory=2200MB) ...
* Preparing Kubernetes v1.16.0 on Docker 25.0.3 ...
- Generating certificates and keys ...
- Booting up control plane ...
- Generating certificates and keys ...
- Booting up control plane ...
-- /stdout --
** stderr **
I0216 09:45:52.316481 18426 out.go:291] Setting OutFile to fd 1 ...
I0216 09:45:52.316867 18426 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0216 09:45:52.316877 18426 out.go:304] Setting ErrFile to fd 2...
I0216 09:45:52.316884 18426 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0216 09:45:52.317201 18426 root.go:338] Updating PATH: /Users/jenkins/minikube-integration/17936-1021/.minikube/bin
I0216 09:45:52.320165 18426 out.go:298] Setting JSON to false
I0216 09:45:52.345401 18426 start.go:129] hostinfo: {"hostname":"MacOS-Agent-1.local","uptime":4523,"bootTime":1708101029,"procs":457,"os":"darwin","platform":"darwin","platformFamily":"Standalone Workstation","platformVersion":"14.3.1","kernelVersion":"23.3.0","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"b7610dcb-1435-5842-8d5a-b2388403fea3"}
W0216 09:45:52.345521 18426 start.go:137] gopshost.Virtualization returned error: not implemented yet
I0216 09:45:52.367685 18426 out.go:177] * [old-k8s-version-356000] minikube v1.32.0 on Darwin 14.3.1
I0216 09:45:52.426495 18426 out.go:177] - MINIKUBE_LOCATION=17936
I0216 09:45:52.426576 18426 notify.go:220] Checking for updates...
I0216 09:45:52.469954 18426 out.go:177] - KUBECONFIG=/Users/jenkins/minikube-integration/17936-1021/kubeconfig
I0216 09:45:52.491290 18426 out.go:177] - MINIKUBE_BIN=out/minikube-darwin-amd64
I0216 09:45:52.512462 18426 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0216 09:45:52.534253 18426 out.go:177] - MINIKUBE_HOME=/Users/jenkins/minikube-integration/17936-1021/.minikube
I0216 09:45:52.555193 18426 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0216 09:45:52.576793 18426 config.go:182] Loaded profile config "kubenet-862000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.28.4
I0216 09:45:52.576930 18426 driver.go:392] Setting default libvirt URI to qemu:///system
I0216 09:45:52.633248 18426 docker.go:122] docker version: linux-25.0.3:Docker Desktop 4.27.2 (137060)
I0216 09:45:52.633415 18426 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0216 09:45:52.746659 18426 info.go:266] docker info: {ID:37b3081a-8cd6-457e-b2a4-79dc82345b06 Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:72 OomKillDisable:false NGoroutines:115 SystemTime:2024-02-16 17:45:52.735654455 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:26 KernelVersion:6.6.12-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServerAddress:
https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:12 MemTotal:6213300224 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=unconfined name
=cgroupns] ProductLicense: Warnings:[WARNING: daemon is not using the default seccomp profile] ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1-desktop.4] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.5-desktop.1] map[Name:debug Path:/Users/jenkins/.docker/cli-plugins/docker-debug SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-debug] ShortDescription:Get a shell into any image or container. Vendor:Docker Inc. Version:0.0.24] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker D
ev Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Manages Docker extensions Vendor:Docker Inc. Version:v0.2.21] map[Name:feedback Path:/Users/jenkins/.docker/cli-plugins/docker-feedback SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-feedback] ShortDescription:Provide feedback, right in your terminal! Vendor:Docker Inc. Version:v1.0.4] map[Name:init Path:/Users/jenkins/.docker/cli-plugins/docker-init SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-init] ShortDescription:Creates Docker-related starter files for your project Vendor:Docker Inc. Version:v1.0.0] map[Name:sbom Path:/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM)
for an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Err:failed to fetch metadata: fork/exec /Users/jenkins/.docker/cli-plugins/docker-scan: no such file or directory Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Docker Scout Vendor:Docker Inc. Version:v1.4.1]] Warnings:<nil>}}
I0216 09:45:52.789314 18426 out.go:177] * Using the docker driver based on user configuration
I0216 09:45:52.810354 18426 start.go:299] selected driver: docker
I0216 09:45:52.810368 18426 start.go:903] validating driver "docker" against <nil>
I0216 09:45:52.810378 18426 start.go:914] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0216 09:45:52.814000 18426 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0216 09:45:52.935066 18426 info.go:266] docker info: {ID:37b3081a-8cd6-457e-b2a4-79dc82345b06 Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:3 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:72 OomKillDisable:false NGoroutines:115 SystemTime:2024-02-16 17:45:52.92486987 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:26 KernelVersion:6.6.12-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServerAddress:h
ttps://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:12 MemTotal:6213300224 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy:http.docker.internal:3128 HTTPSProxy:http.docker.internal:3128 NoProxy:hubproxy.docker.internal Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=unconfined name=
cgroupns] ProductLicense: Warnings:[WARNING: daemon is not using the default seccomp profile] ServerErrors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/Users/jenkins/.docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-buildx] ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1-desktop.4] map[Name:compose Path:/Users/jenkins/.docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-compose] ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.5-desktop.1] map[Name:debug Path:/Users/jenkins/.docker/cli-plugins/docker-debug SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-debug] ShortDescription:Get a shell into any image or container. Vendor:Docker Inc. Version:0.0.24] map[Name:dev Path:/Users/jenkins/.docker/cli-plugins/docker-dev SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-dev] ShortDescription:Docker De
v Environments Vendor:Docker Inc. Version:v0.1.0] map[Name:extension Path:/Users/jenkins/.docker/cli-plugins/docker-extension SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-extension] ShortDescription:Manages Docker extensions Vendor:Docker Inc. Version:v0.2.21] map[Name:feedback Path:/Users/jenkins/.docker/cli-plugins/docker-feedback SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-feedback] ShortDescription:Provide feedback, right in your terminal! Vendor:Docker Inc. Version:v1.0.4] map[Name:init Path:/Users/jenkins/.docker/cli-plugins/docker-init SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-init] ShortDescription:Creates Docker-related starter files for your project Vendor:Docker Inc. Version:v1.0.0] map[Name:sbom Path:/Users/jenkins/.docker/cli-plugins/docker-sbom SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-sbom] ShortDescription:View the packaged-based Software Bill Of Materials (SBOM) f
or an image URL:https://github.com/docker/sbom-cli-plugin Vendor:Anchore Inc. Version:0.6.0] map[Err:failed to fetch metadata: fork/exec /Users/jenkins/.docker/cli-plugins/docker-scan: no such file or directory Name:scan Path:/Users/jenkins/.docker/cli-plugins/docker-scan] map[Name:scout Path:/Users/jenkins/.docker/cli-plugins/docker-scout SchemaVersion:0.1.0 ShadowedPaths:[/usr/local/lib/docker/cli-plugins/docker-scout] ShortDescription:Docker Scout Vendor:Docker Inc. Version:v1.4.1]] Warnings:<nil>}}
I0216 09:45:52.935277 18426 start_flags.go:309] no existing cluster config was found, will generate one from the flags
I0216 09:45:52.935455 18426 start_flags.go:931] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0216 09:45:52.956723 18426 out.go:177] * Using Docker Desktop driver with root privileges
I0216 09:45:52.993913 18426 cni.go:84] Creating CNI manager for ""
I0216 09:45:52.993942 18426 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0216 09:45:52.993954 18426 start_flags.go:323] config:
{Name:old-k8s-version-356000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-356000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local Containe
rRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0216 09:45:53.003212 18426 out.go:177] * Starting control plane node old-k8s-version-356000 in cluster old-k8s-version-356000
I0216 09:45:53.061630 18426 cache.go:121] Beginning downloading kic base image for docker with docker
I0216 09:45:53.082734 18426 out.go:177] * Pulling base image v0.0.42-1708008208-17936 ...
I0216 09:45:53.124609 18426 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0216 09:45:53.124675 18426 image.go:79] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf in local docker daemon
I0216 09:45:53.124739 18426 preload.go:148] Found local preload: /Users/jenkins/minikube-integration/17936-1021/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4
I0216 09:45:53.124787 18426 cache.go:56] Caching tarball of preloaded images
I0216 09:45:53.125101 18426 preload.go:174] Found /Users/jenkins/minikube-integration/17936-1021/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0216 09:45:53.125653 18426 cache.go:59] Finished verifying existence of preloaded tar for v1.16.0 on docker
I0216 09:45:53.126092 18426 profile.go:148] Saving config to /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/config.json ...
I0216 09:45:53.126137 18426 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/config.json: {Name:mkfc113630e7ad903ada6e2a9851d3843fb77e23 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 09:45:53.197786 18426 image.go:83] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf in local docker daemon, skipping pull
I0216 09:45:53.197903 18426 cache.go:144] gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf exists in daemon, skipping load
I0216 09:45:53.197923 18426 cache.go:194] Successfully downloaded all kic artifacts
I0216 09:45:53.198168 18426 start.go:365] acquiring machines lock for old-k8s-version-356000: {Name:mkcbb668d74284a5583a7ae9844b8f225578b58f Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0216 09:45:53.198369 18426 start.go:369] acquired machines lock for "old-k8s-version-356000" in 186.538µs
I0216 09:45:53.198404 18426 start.go:93] Provisioning new machine with config: &{Name:old-k8s-version-356000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-356000 Namespace:default APIServer
Name:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:f
alse DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s} &{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}
I0216 09:45:53.198473 18426 start.go:125] createHost starting for "" (driver="docker")
I0216 09:45:53.242790 18426 out.go:204] * Creating docker container (CPUs=2, Memory=2200MB) ...
I0216 09:45:53.243040 18426 start.go:159] libmachine.API.Create for "old-k8s-version-356000" (driver="docker")
I0216 09:45:53.243072 18426 client.go:168] LocalClient.Create starting
I0216 09:45:53.243195 18426 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/17936-1021/.minikube/certs/ca.pem
I0216 09:45:53.243245 18426 main.go:141] libmachine: Decoding PEM data...
I0216 09:45:53.243262 18426 main.go:141] libmachine: Parsing certificate...
I0216 09:45:53.243319 18426 main.go:141] libmachine: Reading certificate data from /Users/jenkins/minikube-integration/17936-1021/.minikube/certs/cert.pem
I0216 09:45:53.243357 18426 main.go:141] libmachine: Decoding PEM data...
I0216 09:45:53.243377 18426 main.go:141] libmachine: Parsing certificate...
I0216 09:45:53.243822 18426 cli_runner.go:164] Run: docker network inspect old-k8s-version-356000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
W0216 09:45:53.375326 18426 cli_runner.go:211] docker network inspect old-k8s-version-356000 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1
I0216 09:45:53.375456 18426 network_create.go:281] running [docker network inspect old-k8s-version-356000] to gather additional debugging logs...
I0216 09:45:53.375472 18426 cli_runner.go:164] Run: docker network inspect old-k8s-version-356000
W0216 09:45:53.426776 18426 cli_runner.go:211] docker network inspect old-k8s-version-356000 returned with exit code 1
I0216 09:45:53.426805 18426 network_create.go:284] error running [docker network inspect old-k8s-version-356000]: docker network inspect old-k8s-version-356000: exit status 1
stdout:
[]
stderr:
Error response from daemon: network old-k8s-version-356000 not found
I0216 09:45:53.426816 18426 network_create.go:286] output of [docker network inspect old-k8s-version-356000]: -- stdout --
[]
-- /stdout --
** stderr **
Error response from daemon: network old-k8s-version-356000 not found
** /stderr **
I0216 09:45:53.426967 18426 cli_runner.go:164] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0216 09:45:53.478271 18426 network.go:210] skipping subnet 192.168.49.0/24 that is reserved: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0216 09:45:53.479894 18426 network.go:210] skipping subnet 192.168.58.0/24 that is reserved: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0216 09:45:53.481479 18426 network.go:210] skipping subnet 192.168.67.0/24 that is reserved: &{IP:192.168.67.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.67.0/24 Gateway:192.168.67.1 ClientMin:192.168.67.2 ClientMax:192.168.67.254 Broadcast:192.168.67.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:<nil>}
I0216 09:45:53.481832 18426 network.go:207] using free private subnet 192.168.76.0/24: &{IP:192.168.76.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.76.0/24 Gateway:192.168.76.1 ClientMin:192.168.76.2 ClientMax:192.168.76.254 Broadcast:192.168.76.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc002368540}
I0216 09:45:53.481851 18426 network_create.go:124] attempt to create docker network old-k8s-version-356000 192.168.76.0/24 with gateway 192.168.76.1 and MTU of 65535 ...
I0216 09:45:53.481917 18426 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.76.0/24 --gateway=192.168.76.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-356000 old-k8s-version-356000
I0216 09:45:53.568543 18426 network_create.go:108] docker network old-k8s-version-356000 192.168.76.0/24 created
I0216 09:45:53.568581 18426 kic.go:121] calculated static IP "192.168.76.2" for the "old-k8s-version-356000" container
I0216 09:45:53.568698 18426 cli_runner.go:164] Run: docker ps -a --format {{.Names}}
I0216 09:45:53.620288 18426 cli_runner.go:164] Run: docker volume create old-k8s-version-356000 --label name.minikube.sigs.k8s.io=old-k8s-version-356000 --label created_by.minikube.sigs.k8s.io=true
I0216 09:45:53.671601 18426 oci.go:103] Successfully created a docker volume old-k8s-version-356000
I0216 09:45:53.671710 18426 cli_runner.go:164] Run: docker run --rm --name old-k8s-version-356000-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-356000 --entrypoint /usr/bin/test -v old-k8s-version-356000:/var gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf -d /var/lib
I0216 09:45:54.138024 18426 oci.go:107] Successfully prepared a docker volume old-k8s-version-356000
I0216 09:45:54.138079 18426 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0216 09:45:54.138092 18426 kic.go:194] Starting extracting preloaded images to volume ...
I0216 09:45:54.138187 18426 cli_runner.go:164] Run: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/17936-1021/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-356000:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf -I lz4 -xf /preloaded.tar -C /extractDir
I0216 09:45:56.343354 18426 cli_runner.go:217] Completed: docker run --rm --entrypoint /usr/bin/tar -v /Users/jenkins/minikube-integration/17936-1021/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-356000:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf -I lz4 -xf /preloaded.tar -C /extractDir: (2.205023157s)
I0216 09:45:56.343385 18426 kic.go:203] duration metric: took 2.205249 seconds to extract preloaded images to volume
I0216 09:45:56.343548 18426 cli_runner.go:164] Run: docker info --format "'{{json .SecurityOptions}}'"
I0216 09:45:56.472077 18426 cli_runner.go:164] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname old-k8s-version-356000 --name old-k8s-version-356000 --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-356000 --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=old-k8s-version-356000 --network old-k8s-version-356000 --ip 192.168.76.2 --volume old-k8s-version-356000:/var --security-opt apparmor=unconfined --memory=2200mb --memory-swap=2200mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf
I0216 09:45:56.795052 18426 cli_runner.go:164] Run: docker container inspect old-k8s-version-356000 --format={{.State.Running}}
I0216 09:45:56.857179 18426 cli_runner.go:164] Run: docker container inspect old-k8s-version-356000 --format={{.State.Status}}
I0216 09:45:56.917226 18426 cli_runner.go:164] Run: docker exec old-k8s-version-356000 stat /var/lib/dpkg/alternatives/iptables
I0216 09:45:57.027377 18426 oci.go:144] the created container "old-k8s-version-356000" has a running status.
I0216 09:45:57.027418 18426 kic.go:225] Creating ssh key for kic: /Users/jenkins/minikube-integration/17936-1021/.minikube/machines/old-k8s-version-356000/id_rsa...
I0216 09:45:57.117840 18426 kic_runner.go:191] docker (temp): /Users/jenkins/minikube-integration/17936-1021/.minikube/machines/old-k8s-version-356000/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0216 09:45:57.185603 18426 cli_runner.go:164] Run: docker container inspect old-k8s-version-356000 --format={{.State.Status}}
I0216 09:45:57.245021 18426 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0216 09:45:57.245056 18426 kic_runner.go:114] Args: [docker exec --privileged old-k8s-version-356000 chown docker:docker /home/docker/.ssh/authorized_keys]
I0216 09:45:57.366977 18426 cli_runner.go:164] Run: docker container inspect old-k8s-version-356000 --format={{.State.Status}}
I0216 09:45:57.427798 18426 machine.go:88] provisioning docker machine ...
I0216 09:45:57.427857 18426 ubuntu.go:169] provisioning hostname "old-k8s-version-356000"
I0216 09:45:57.427980 18426 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-356000
I0216 09:45:57.486888 18426 main.go:141] libmachine: Using SSH client type: native
I0216 09:45:57.487207 18426 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1407080] 0x1409d60 <nil> [] 0s} 127.0.0.1 53808 <nil> <nil>}
I0216 09:45:57.487219 18426 main.go:141] libmachine: About to run SSH command:
sudo hostname old-k8s-version-356000 && echo "old-k8s-version-356000" | sudo tee /etc/hostname
I0216 09:45:57.647650 18426 main.go:141] libmachine: SSH cmd err, output: <nil>: old-k8s-version-356000
I0216 09:45:57.647763 18426 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-356000
I0216 09:45:57.700626 18426 main.go:141] libmachine: Using SSH client type: native
I0216 09:45:57.700914 18426 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1407080] 0x1409d60 <nil> [] 0s} 127.0.0.1 53808 <nil> <nil>}
I0216 09:45:57.700932 18426 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sold-k8s-version-356000' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 old-k8s-version-356000/g' /etc/hosts;
else
echo '127.0.1.1 old-k8s-version-356000' | sudo tee -a /etc/hosts;
fi
fi
I0216 09:45:57.837383 18426 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0216 09:45:57.837419 18426 ubuntu.go:175] set auth options {CertDir:/Users/jenkins/minikube-integration/17936-1021/.minikube CaCertPath:/Users/jenkins/minikube-integration/17936-1021/.minikube/certs/ca.pem CaPrivateKeyPath:/Users/jenkins/minikube-integration/17936-1021/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/Users/jenkins/minikube-integration/17936-1021/.minikube/machines/server.pem ServerKeyPath:/Users/jenkins/minikube-integration/17936-1021/.minikube/machines/server-key.pem ClientKeyPath:/Users/jenkins/minikube-integration/17936-1021/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/Users/jenkins/minikube-integration/17936-1021/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/Users/jenkins/minikube-integration/17936-1021/.minikube}
I0216 09:45:57.837442 18426 ubuntu.go:177] setting up certificates
I0216 09:45:57.837455 18426 provision.go:83] configureAuth start
I0216 09:45:57.837556 18426 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-356000
I0216 09:45:57.890257 18426 provision.go:138] copyHostCerts
I0216 09:45:57.890373 18426 exec_runner.go:144] found /Users/jenkins/minikube-integration/17936-1021/.minikube/ca.pem, removing ...
I0216 09:45:57.890386 18426 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/17936-1021/.minikube/ca.pem
I0216 09:45:57.890522 18426 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/17936-1021/.minikube/certs/ca.pem --> /Users/jenkins/minikube-integration/17936-1021/.minikube/ca.pem (1082 bytes)
I0216 09:45:57.890766 18426 exec_runner.go:144] found /Users/jenkins/minikube-integration/17936-1021/.minikube/cert.pem, removing ...
I0216 09:45:57.890773 18426 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/17936-1021/.minikube/cert.pem
I0216 09:45:57.890850 18426 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/17936-1021/.minikube/certs/cert.pem --> /Users/jenkins/minikube-integration/17936-1021/.minikube/cert.pem (1123 bytes)
I0216 09:45:57.891029 18426 exec_runner.go:144] found /Users/jenkins/minikube-integration/17936-1021/.minikube/key.pem, removing ...
I0216 09:45:57.891037 18426 exec_runner.go:203] rm: /Users/jenkins/minikube-integration/17936-1021/.minikube/key.pem
I0216 09:45:57.891111 18426 exec_runner.go:151] cp: /Users/jenkins/minikube-integration/17936-1021/.minikube/certs/key.pem --> /Users/jenkins/minikube-integration/17936-1021/.minikube/key.pem (1675 bytes)
I0216 09:45:57.891279 18426 provision.go:112] generating server cert: /Users/jenkins/minikube-integration/17936-1021/.minikube/machines/server.pem ca-key=/Users/jenkins/minikube-integration/17936-1021/.minikube/certs/ca.pem private-key=/Users/jenkins/minikube-integration/17936-1021/.minikube/certs/ca-key.pem org=jenkins.old-k8s-version-356000 san=[192.168.76.2 127.0.0.1 localhost 127.0.0.1 minikube old-k8s-version-356000]
I0216 09:45:57.956629 18426 provision.go:172] copyRemoteCerts
I0216 09:45:57.956705 18426 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0216 09:45:57.956768 18426 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-356000
I0216 09:45:58.010018 18426 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:53808 SSHKeyPath:/Users/jenkins/minikube-integration/17936-1021/.minikube/machines/old-k8s-version-356000/id_rsa Username:docker}
I0216 09:45:58.110967 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1082 bytes)
I0216 09:45:58.151852 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/machines/server.pem --> /etc/docker/server.pem (1241 bytes)
I0216 09:45:58.193106 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1679 bytes)
I0216 09:45:58.233909 18426 provision.go:86] duration metric: configureAuth took 396.430861ms
I0216 09:45:58.233925 18426 ubuntu.go:193] setting minikube options for container-runtime
I0216 09:45:58.234062 18426 config.go:182] Loaded profile config "old-k8s-version-356000": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.16.0
I0216 09:45:58.234132 18426 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-356000
I0216 09:45:58.289844 18426 main.go:141] libmachine: Using SSH client type: native
I0216 09:45:58.290164 18426 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1407080] 0x1409d60 <nil> [] 0s} 127.0.0.1 53808 <nil> <nil>}
I0216 09:45:58.290183 18426 main.go:141] libmachine: About to run SSH command:
df --output=fstype / | tail -n 1
I0216 09:45:58.427624 18426 main.go:141] libmachine: SSH cmd err, output: <nil>: overlay
I0216 09:45:58.427642 18426 ubuntu.go:71] root file system type: overlay
I0216 09:45:58.427757 18426 provision.go:309] Updating docker unit: /lib/systemd/system/docker.service ...
I0216 09:45:58.427843 18426 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-356000
I0216 09:45:58.480583 18426 main.go:141] libmachine: Using SSH client type: native
I0216 09:45:58.480912 18426 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1407080] 0x1409d60 <nil> [] 0s} 127.0.0.1 53808 <nil> <nil>}
I0216 09:45:58.480960 18426 main.go:141] libmachine: About to run SSH command:
sudo mkdir -p /lib/systemd/system && printf %s "[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP \$MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service.new
I0216 09:45:58.639357 18426 main.go:141] libmachine: SSH cmd err, output: <nil>: [Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
I0216 09:45:58.639469 18426 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-356000
I0216 09:45:58.693191 18426 main.go:141] libmachine: Using SSH client type: native
I0216 09:45:58.693484 18426 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x1407080] 0x1409d60 <nil> [] 0s} 127.0.0.1 53808 <nil> <nil>}
I0216 09:45:58.693497 18426 main.go:141] libmachine: About to run SSH command:
sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
I0216 09:45:59.356804 18426 main.go:141] libmachine: SSH cmd err, output: <nil>: --- /lib/systemd/system/docker.service 2024-02-06 21:12:51.000000000 +0000
+++ /lib/systemd/system/docker.service.new 2024-02-16 17:45:58.634389397 +0000
@@ -1,46 +1,49 @@
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
-After=network-online.target docker.socket firewalld.service containerd.service time-set.target
-Wants=network-online.target containerd.service
+BindsTo=containerd.service
+After=network-online.target firewalld.service containerd.service
+Wants=network-online.target
Requires=docker.socket
+StartLimitBurst=3
+StartLimitIntervalSec=60
[Service]
Type=notify
-# the default is not to use systemd for cgroups because the delegate issues still
-# exists and systemd currently does not support the cgroup feature set required
-# for containers run by docker
-ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
-ExecReload=/bin/kill -s HUP $MAINPID
-TimeoutStartSec=0
-RestartSec=2
-Restart=always
+Restart=on-failure
-# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
-# Both the old, and new location are accepted by systemd 229 and up, so using the old location
-# to make them work for either version of systemd.
-StartLimitBurst=3
-# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
-# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
-# this option work for either version of systemd.
-StartLimitInterval=60s
+
+# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
+# The base configuration already specifies an 'ExecStart=...' command. The first directive
+# here is to clear out that command inherited from the base configuration. Without this,
+# the command from the base configuration and the command specified here are treated as
+# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
+# will catch this invalid input and refuse to start the service with an error like:
+# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
+
+# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
+# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
+ExecStart=
+ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
+ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
+LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
-# Comment TasksMax if your systemd version does not support it.
-# Only systemd 226 and above support this option.
+# Uncomment TasksMax if your systemd version supports it.
+# Only systemd 226 and above support this version.
TasksMax=infinity
+TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
-OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
I0216 09:45:59.356823 18426 machine.go:91] provisioned docker machine in 1.928951734s
I0216 09:45:59.356830 18426 client.go:171] LocalClient.Create took 6.113632302s
I0216 09:45:59.356850 18426 start.go:167] duration metric: libmachine.API.Create for "old-k8s-version-356000" took 6.11369231s
I0216 09:45:59.356861 18426 start.go:300] post-start starting for "old-k8s-version-356000" (driver="docker")
I0216 09:45:59.356869 18426 start.go:329] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0216 09:45:59.356938 18426 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0216 09:45:59.356999 18426 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-356000
I0216 09:45:59.410326 18426 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:53808 SSHKeyPath:/Users/jenkins/minikube-integration/17936-1021/.minikube/machines/old-k8s-version-356000/id_rsa Username:docker}
I0216 09:45:59.513617 18426 ssh_runner.go:195] Run: cat /etc/os-release
I0216 09:45:59.517619 18426 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0216 09:45:59.517647 18426 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0216 09:45:59.517655 18426 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0216 09:45:59.517661 18426 info.go:137] Remote host: Ubuntu 22.04.3 LTS
I0216 09:45:59.517672 18426 filesync.go:126] Scanning /Users/jenkins/minikube-integration/17936-1021/.minikube/addons for local assets ...
I0216 09:45:59.517778 18426 filesync.go:126] Scanning /Users/jenkins/minikube-integration/17936-1021/.minikube/files for local assets ...
I0216 09:45:59.517967 18426 filesync.go:149] local asset: /Users/jenkins/minikube-integration/17936-1021/.minikube/files/etc/ssl/certs/21512.pem -> 21512.pem in /etc/ssl/certs
I0216 09:45:59.518167 18426 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0216 09:45:59.533673 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/files/etc/ssl/certs/21512.pem --> /etc/ssl/certs/21512.pem (1708 bytes)
I0216 09:45:59.574739 18426 start.go:303] post-start completed in 217.862189ms
I0216 09:45:59.575328 18426 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-356000
I0216 09:45:59.626216 18426 profile.go:148] Saving config to /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/config.json ...
I0216 09:45:59.626704 18426 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0216 09:45:59.626763 18426 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-356000
I0216 09:45:59.678570 18426 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:53808 SSHKeyPath:/Users/jenkins/minikube-integration/17936-1021/.minikube/machines/old-k8s-version-356000/id_rsa Username:docker}
I0216 09:45:59.769477 18426 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0216 09:45:59.775092 18426 start.go:128] duration metric: createHost completed in 6.576470429s
I0216 09:45:59.775113 18426 start.go:83] releasing machines lock for "old-k8s-version-356000", held for 6.576604434s
I0216 09:45:59.775215 18426 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-356000
I0216 09:45:59.833337 18426 ssh_runner.go:195] Run: cat /version.json
I0216 09:45:59.833343 18426 ssh_runner.go:195] Run: curl -sS -m 2 https://registry.k8s.io/
I0216 09:45:59.833415 18426 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-356000
I0216 09:45:59.833468 18426 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-356000
I0216 09:45:59.894814 18426 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:53808 SSHKeyPath:/Users/jenkins/minikube-integration/17936-1021/.minikube/machines/old-k8s-version-356000/id_rsa Username:docker}
I0216 09:45:59.894840 18426 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:53808 SSHKeyPath:/Users/jenkins/minikube-integration/17936-1021/.minikube/machines/old-k8s-version-356000/id_rsa Username:docker}
I0216 09:46:00.092612 18426 ssh_runner.go:195] Run: systemctl --version
I0216 09:46:00.097398 18426 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
I0216 09:46:00.102514 18426 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *loopback.conf* -not -name *.mk_disabled -exec sh -c "grep -q loopback {} && ( grep -q name {} || sudo sed -i '/"type": "loopback"/i \ \ \ \ "name": "loopback",' {} ) && sudo sed -i 's|"cniVersion": ".*"|"cniVersion": "1.0.0"|g' {}" ;
I0216 09:46:00.143185 18426 cni.go:230] loopback cni configuration patched: "/etc/cni/net.d/*loopback.conf*" found
I0216 09:46:00.143257 18426 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *bridge* -not -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e '/"dst": ".*:.*"/d' -e 's|^(.*)"dst": (.*)[,*]$|\1"dst": \2|g' -e '/"subnet": ".*:.*"/d' -e 's|^(.*)"subnet": ".*"(.*)[,*]$|\1"subnet": "10.244.0.0/16"\2|g' {}" ;
I0216 09:46:00.171242 18426 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e 's|^(.*)"subnet": ".*"(.*)$|\1"subnet": "10.244.0.0/16"\2|g' -e 's|^(.*)"gateway": ".*"(.*)$|\1"gateway": "10.244.0.1"\2|g' {}" ;
I0216 09:46:00.199884 18426 cni.go:308] configured [/etc/cni/net.d/100-crio-bridge.conf, /etc/cni/net.d/87-podman-bridge.conflist] bridge cni config(s)
I0216 09:46:00.199902 18426 start.go:475] detecting cgroup driver to use...
I0216 09:46:00.199919 18426 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0216 09:46:00.200016 18426 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0216 09:46:00.227593 18426 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.1"|' /etc/containerd/config.toml"
I0216 09:46:00.243923 18426 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I0216 09:46:00.259850 18426 containerd.go:146] configuring containerd to use "cgroupfs" as cgroup driver...
I0216 09:46:00.259905 18426 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = false|g' /etc/containerd/config.toml"
I0216 09:46:00.277152 18426 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0216 09:46:00.295187 18426 ssh_runner.go:195] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I0216 09:46:00.312201 18426 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0216 09:46:00.334101 18426 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0216 09:46:00.351121 18426 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I0216 09:46:00.367311 18426 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0216 09:46:00.383090 18426 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0216 09:46:00.398215 18426 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0216 09:46:00.463708 18426 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0216 09:46:00.556201 18426 start.go:475] detecting cgroup driver to use...
I0216 09:46:00.556222 18426 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0216 09:46:00.556298 18426 ssh_runner.go:195] Run: sudo systemctl cat docker.service
I0216 09:46:00.576310 18426 cruntime.go:279] skipping containerd shutdown because we are bound to it
I0216 09:46:00.576414 18426 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0216 09:46:00.596454 18426 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///var/run/dockershim.sock
" | sudo tee /etc/crictl.yaml"
I0216 09:46:00.630643 18426 ssh_runner.go:195] Run: which cri-dockerd
I0216 09:46:00.635483 18426 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/cri-docker.service.d
I0216 09:46:00.651849 18426 ssh_runner.go:362] scp memory --> /etc/systemd/system/cri-docker.service.d/10-cni.conf (189 bytes)
I0216 09:46:00.682029 18426 ssh_runner.go:195] Run: sudo systemctl unmask docker.service
I0216 09:46:00.753293 18426 ssh_runner.go:195] Run: sudo systemctl enable docker.socket
I0216 09:46:00.856329 18426 docker.go:574] configuring docker to use "cgroupfs" as cgroup driver...
I0216 09:46:00.856426 18426 ssh_runner.go:362] scp memory --> /etc/docker/daemon.json (130 bytes)
I0216 09:46:00.885904 18426 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0216 09:46:00.950945 18426 ssh_runner.go:195] Run: sudo systemctl restart docker
I0216 09:46:01.207156 18426 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0216 09:46:01.232320 18426 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0216 09:46:01.301971 18426 out.go:204] * Preparing Kubernetes v1.16.0 on Docker 25.0.3 ...
I0216 09:46:01.302079 18426 cli_runner.go:164] Run: docker exec -t old-k8s-version-356000 dig +short host.docker.internal
I0216 09:46:01.427082 18426 network.go:96] got host ip for mount in container by digging dns: 192.168.65.254
I0216 09:46:01.427174 18426 ssh_runner.go:195] Run: grep 192.168.65.254 host.minikube.internal$ /etc/hosts
I0216 09:46:01.431782 18426 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.65.254 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0216 09:46:01.449333 18426 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "8443/tcp") 0).HostPort}}'" old-k8s-version-356000
I0216 09:46:01.502090 18426 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0216 09:46:01.502174 18426 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0216 09:46:01.523335 18426 docker.go:685] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0216 09:46:01.523349 18426 docker.go:691] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0216 09:46:01.523421 18426 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0216 09:46:01.538593 18426 ssh_runner.go:195] Run: which lz4
I0216 09:46:01.543388 18426 ssh_runner.go:195] Run: stat -c "%s %y" /preloaded.tar.lz4
I0216 09:46:01.547613 18426 ssh_runner.go:352] existence check for /preloaded.tar.lz4: stat -c "%s %y" /preloaded.tar.lz4: Process exited with status 1
stdout:
stderr:
stat: cannot statx '/preloaded.tar.lz4': No such file or directory
I0216 09:46:01.547635 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 --> /preloaded.tar.lz4 (369789069 bytes)
I0216 09:46:07.790983 18426 docker.go:649] Took 6.247516 seconds to copy over tarball
I0216 09:46:07.791070 18426 ssh_runner.go:195] Run: sudo tar --xattrs --xattrs-include security.capability -I lz4 -C /var -xf /preloaded.tar.lz4
I0216 09:46:09.515434 18426 ssh_runner.go:235] Completed: sudo tar --xattrs --xattrs-include security.capability -I lz4 -C /var -xf /preloaded.tar.lz4: (1.724312958s)
I0216 09:46:09.515449 18426 ssh_runner.go:146] rm: /preloaded.tar.lz4
I0216 09:46:09.567297 18426 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0216 09:46:09.582704 18426 ssh_runner.go:362] scp memory --> /var/lib/docker/image/overlay2/repositories.json (2499 bytes)
I0216 09:46:09.615348 18426 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0216 09:46:09.678987 18426 ssh_runner.go:195] Run: sudo systemctl restart docker
I0216 09:46:10.325388 18426 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0216 09:46:10.346668 18426 docker.go:685] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0216 09:46:10.346680 18426 docker.go:691] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0216 09:46:10.346690 18426 cache_images.go:88] LoadImages start: [registry.k8s.io/kube-apiserver:v1.16.0 registry.k8s.io/kube-controller-manager:v1.16.0 registry.k8s.io/kube-scheduler:v1.16.0 registry.k8s.io/kube-proxy:v1.16.0 registry.k8s.io/pause:3.1 registry.k8s.io/etcd:3.3.15-0 registry.k8s.io/coredns:1.6.2 gcr.io/k8s-minikube/storage-provisioner:v5]
I0216 09:46:10.351745 18426 image.go:134] retrieving image: registry.k8s.io/kube-controller-manager:v1.16.0
I0216 09:46:10.351930 18426 image.go:134] retrieving image: registry.k8s.io/coredns:1.6.2
I0216 09:46:10.352151 18426 image.go:134] retrieving image: gcr.io/k8s-minikube/storage-provisioner:v5
I0216 09:46:10.352448 18426 image.go:134] retrieving image: registry.k8s.io/kube-apiserver:v1.16.0
I0216 09:46:10.353303 18426 image.go:134] retrieving image: registry.k8s.io/kube-proxy:v1.16.0
I0216 09:46:10.353805 18426 image.go:134] retrieving image: registry.k8s.io/etcd:3.3.15-0
I0216 09:46:10.353895 18426 image.go:134] retrieving image: registry.k8s.io/kube-scheduler:v1.16.0
I0216 09:46:10.353911 18426 image.go:134] retrieving image: registry.k8s.io/pause:3.1
I0216 09:46:10.355855 18426 image.go:177] daemon lookup for registry.k8s.io/coredns:1.6.2: Error response from daemon: No such image: registry.k8s.io/coredns:1.6.2
I0216 09:46:10.358680 18426 image.go:177] daemon lookup for gcr.io/k8s-minikube/storage-provisioner:v5: Error response from daemon: No such image: gcr.io/k8s-minikube/storage-provisioner:v5
I0216 09:46:10.358854 18426 image.go:177] daemon lookup for registry.k8s.io/kube-proxy:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-proxy:v1.16.0
I0216 09:46:10.358956 18426 image.go:177] daemon lookup for registry.k8s.io/kube-controller-manager:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-controller-manager:v1.16.0
I0216 09:46:10.359114 18426 image.go:177] daemon lookup for registry.k8s.io/kube-apiserver:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-apiserver:v1.16.0
I0216 09:46:10.361634 18426 image.go:177] daemon lookup for registry.k8s.io/etcd:3.3.15-0: Error response from daemon: No such image: registry.k8s.io/etcd:3.3.15-0
I0216 09:46:10.362090 18426 image.go:177] daemon lookup for registry.k8s.io/pause:3.1: Error response from daemon: No such image: registry.k8s.io/pause:3.1
I0216 09:46:10.362166 18426 image.go:177] daemon lookup for registry.k8s.io/kube-scheduler:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-scheduler:v1.16.0
I0216 09:46:12.251831 18426 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-proxy:v1.16.0
I0216 09:46:12.278620 18426 cache_images.go:116] "registry.k8s.io/kube-proxy:v1.16.0" needs transfer: "registry.k8s.io/kube-proxy:v1.16.0" does not exist at hash "c21b0c7400f988db4777858edd13b6d3930d62d7ccf026d2415485a52037f384" in container runtime
I0216 09:46:12.278661 18426 docker.go:337] Removing image: registry.k8s.io/kube-proxy:v1.16.0
I0216 09:46:12.278729 18426 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-proxy:v1.16.0
I0216 09:46:12.297991 18426 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17936-1021/.minikube/cache/images/amd64/registry.k8s.io/kube-proxy_v1.16.0
I0216 09:46:12.318032 18426 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-apiserver:v1.16.0
I0216 09:46:12.356956 18426 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-controller-manager:v1.16.0
I0216 09:46:12.362074 18426 cache_images.go:116] "registry.k8s.io/kube-apiserver:v1.16.0" needs transfer: "registry.k8s.io/kube-apiserver:v1.16.0" does not exist at hash "b305571ca60a5a7818bda47da122683d75e8a1907475681ee8b1efbd06bff12e" in container runtime
I0216 09:46:12.362103 18426 docker.go:337] Removing image: registry.k8s.io/kube-apiserver:v1.16.0
I0216 09:46:12.362168 18426 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-apiserver:v1.16.0
I0216 09:46:12.379322 18426 cache_images.go:116] "registry.k8s.io/kube-controller-manager:v1.16.0" needs transfer: "registry.k8s.io/kube-controller-manager:v1.16.0" does not exist at hash "06a629a7e51cdcc81a5ed6a3e6650348312f20c954ac52ee489a023628ec9c7d" in container runtime
I0216 09:46:12.379356 18426 docker.go:337] Removing image: registry.k8s.io/kube-controller-manager:v1.16.0
I0216 09:46:12.379436 18426 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-controller-manager:v1.16.0
I0216 09:46:12.384375 18426 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/coredns:1.6.2
I0216 09:46:12.386412 18426 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17936-1021/.minikube/cache/images/amd64/registry.k8s.io/kube-apiserver_v1.16.0
I0216 09:46:12.394961 18426 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/pause:3.1
I0216 09:46:12.401759 18426 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17936-1021/.minikube/cache/images/amd64/registry.k8s.io/kube-controller-manager_v1.16.0
I0216 09:46:12.406407 18426 cache_images.go:116] "registry.k8s.io/coredns:1.6.2" needs transfer: "registry.k8s.io/coredns:1.6.2" does not exist at hash "bf261d157914477ee1a5969d28ec687f3fbfc9fb5a664b22df78e57023b0e03b" in container runtime
I0216 09:46:12.406432 18426 docker.go:337] Removing image: registry.k8s.io/coredns:1.6.2
I0216 09:46:12.406494 18426 ssh_runner.go:195] Run: docker rmi registry.k8s.io/coredns:1.6.2
I0216 09:46:12.411327 18426 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-scheduler:v1.16.0
I0216 09:46:12.418419 18426 cache_images.go:116] "registry.k8s.io/pause:3.1" needs transfer: "registry.k8s.io/pause:3.1" does not exist at hash "da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e" in container runtime
I0216 09:46:12.418458 18426 docker.go:337] Removing image: registry.k8s.io/pause:3.1
I0216 09:46:12.418529 18426 ssh_runner.go:195] Run: docker rmi registry.k8s.io/pause:3.1
I0216 09:46:12.419070 18426 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/etcd:3.3.15-0
I0216 09:46:12.433610 18426 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17936-1021/.minikube/cache/images/amd64/registry.k8s.io/coredns_1.6.2
I0216 09:46:12.442124 18426 cache_images.go:116] "registry.k8s.io/kube-scheduler:v1.16.0" needs transfer: "registry.k8s.io/kube-scheduler:v1.16.0" does not exist at hash "301ddc62b80b16315d3c2653cf3888370394277afb3187614cfa20edc352ca0a" in container runtime
I0216 09:46:12.442184 18426 docker.go:337] Removing image: registry.k8s.io/kube-scheduler:v1.16.0
I0216 09:46:12.442268 18426 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-scheduler:v1.16.0
I0216 09:46:12.450077 18426 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17936-1021/.minikube/cache/images/amd64/registry.k8s.io/pause_3.1
I0216 09:46:12.451233 18426 cache_images.go:116] "registry.k8s.io/etcd:3.3.15-0" needs transfer: "registry.k8s.io/etcd:3.3.15-0" does not exist at hash "b2756210eeabf84f3221da9959e9483f3919dc2aaab4cd45e7cd072fcbde27ed" in container runtime
I0216 09:46:12.451258 18426 docker.go:337] Removing image: registry.k8s.io/etcd:3.3.15-0
I0216 09:46:12.451328 18426 ssh_runner.go:195] Run: docker rmi registry.k8s.io/etcd:3.3.15-0
I0216 09:46:12.508668 18426 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17936-1021/.minikube/cache/images/amd64/registry.k8s.io/kube-scheduler_v1.16.0
I0216 09:46:12.514457 18426 cache_images.go:286] Loading image from: /Users/jenkins/minikube-integration/17936-1021/.minikube/cache/images/amd64/registry.k8s.io/etcd_3.3.15-0
I0216 09:46:12.717186 18426 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} gcr.io/k8s-minikube/storage-provisioner:v5
I0216 09:46:12.738469 18426 cache_images.go:92] LoadImages completed in 2.391719777s
W0216 09:46:12.738524 18426 out.go:239] X Unable to load cached images: loading cached images: stat /Users/jenkins/minikube-integration/17936-1021/.minikube/cache/images/amd64/registry.k8s.io/kube-proxy_v1.16.0: no such file or directory
X Unable to load cached images: loading cached images: stat /Users/jenkins/minikube-integration/17936-1021/.minikube/cache/images/amd64/registry.k8s.io/kube-proxy_v1.16.0: no such file or directory
I0216 09:46:12.738600 18426 ssh_runner.go:195] Run: docker info --format {{.CgroupDriver}}
I0216 09:46:12.791434 18426 cni.go:84] Creating CNI manager for ""
I0216 09:46:12.791453 18426 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0216 09:46:12.791466 18426 kubeadm.go:87] Using pod CIDR: 10.244.0.0/16
I0216 09:46:12.791483 18426 kubeadm.go:176] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.76.2 APIServerPort:8443 KubernetesVersion:v1.16.0 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:old-k8s-version-356000 NodeName:old-k8s-version-356000 DNSDomain:cluster.local CRISocket:/var/run/dockershim.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.76.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.76.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticP
odPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[hairpinMode:hairpin-veth runtimeRequestTimeout:15m] PrependCriSocketUnix:false}
I0216 09:46:12.791570 18426 kubeadm.go:181] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.76.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: "old-k8s-version-356000"
kubeletExtraArgs:
node-ip: 192.168.76.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.76.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: old-k8s-version-356000
controlPlaneEndpoint: control-plane.minikube.internal:8443
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
listen-metrics-urls: http://127.0.0.1:2381,http://192.168.76.2:2381
kubernetesVersion: v1.16.0
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0216 09:46:12.791622 18426 kubeadm.go:976] kubelet [Unit]
Wants=docker.socket
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.16.0/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=old-k8s-version-356000 --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.76.2
[Install]
config:
{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-356000 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:}
I0216 09:46:12.791712 18426 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.16.0
I0216 09:46:12.806824 18426 binaries.go:44] Found k8s binaries, skipping transfer
I0216 09:46:12.806895 18426 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0216 09:46:12.821340 18426 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (348 bytes)
I0216 09:46:12.849560 18426 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0216 09:46:12.879110 18426 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2174 bytes)
I0216 09:46:12.911998 18426 ssh_runner.go:195] Run: grep 192.168.76.2 control-plane.minikube.internal$ /etc/hosts
I0216 09:46:12.917393 18426 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.76.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0216 09:46:12.944213 18426 certs.go:56] Setting up /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000 for IP: 192.168.76.2
I0216 09:46:12.944268 18426 certs.go:190] acquiring lock for shared ca certs: {Name:mk8795f926ccc5dd497b243df5a2c158b5c5b28d Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 09:46:12.944540 18426 certs.go:199] skipping minikubeCA CA generation: /Users/jenkins/minikube-integration/17936-1021/.minikube/ca.key
I0216 09:46:12.944646 18426 certs.go:199] skipping proxyClientCA CA generation: /Users/jenkins/minikube-integration/17936-1021/.minikube/proxy-client-ca.key
I0216 09:46:12.944698 18426 certs.go:319] generating minikube-user signed cert: /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/client.key
I0216 09:46:12.944711 18426 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/client.crt with IP's: []
I0216 09:46:13.048986 18426 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/client.crt ...
I0216 09:46:13.049001 18426 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/client.crt: {Name:mka7fc7da421879d5f99f6bc7a1e1055a9f679a4 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 09:46:13.049301 18426 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/client.key ...
I0216 09:46:13.049325 18426 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/client.key: {Name:mk6dbf4477e491adb529d92a6d504aabd983d6e6 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 09:46:13.049547 18426 certs.go:319] generating minikube signed cert: /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/apiserver.key.31bdca25
I0216 09:46:13.049561 18426 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/apiserver.crt.31bdca25 with IP's: [192.168.76.2 10.96.0.1 127.0.0.1 10.0.0.1]
I0216 09:46:13.186832 18426 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/apiserver.crt.31bdca25 ...
I0216 09:46:13.186846 18426 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/apiserver.crt.31bdca25: {Name:mk293dd0e774fc295fa25febb20d6dc0842fee35 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 09:46:13.187150 18426 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/apiserver.key.31bdca25 ...
I0216 09:46:13.187160 18426 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/apiserver.key.31bdca25: {Name:mk96fc7d316f9527d3f8571aa727fa0dab5f1c06 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 09:46:13.187361 18426 certs.go:337] copying /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/apiserver.crt.31bdca25 -> /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/apiserver.crt
I0216 09:46:13.187535 18426 certs.go:341] copying /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/apiserver.key.31bdca25 -> /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/apiserver.key
I0216 09:46:13.187700 18426 certs.go:319] generating aggregator signed cert: /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/proxy-client.key
I0216 09:46:13.187721 18426 crypto.go:68] Generating cert /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/proxy-client.crt with IP's: []
I0216 09:46:13.315751 18426 crypto.go:156] Writing cert to /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/proxy-client.crt ...
I0216 09:46:13.315764 18426 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/proxy-client.crt: {Name:mkf231c5ceb7687b1415f9d80d570c0f7d7c4acc Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 09:46:13.341124 18426 crypto.go:164] Writing key to /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/proxy-client.key ...
I0216 09:46:13.341155 18426 lock.go:35] WriteFile acquiring /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/proxy-client.key: {Name:mkb2f4eaf4664e8f738301025f0fdeb79ddb9bfc Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 09:46:13.364309 18426 certs.go:437] found cert: /Users/jenkins/minikube-integration/17936-1021/.minikube/certs/Users/jenkins/minikube-integration/17936-1021/.minikube/certs/2151.pem (1338 bytes)
W0216 09:46:13.364396 18426 certs.go:433] ignoring /Users/jenkins/minikube-integration/17936-1021/.minikube/certs/Users/jenkins/minikube-integration/17936-1021/.minikube/certs/2151_empty.pem, impossibly tiny 0 bytes
I0216 09:46:13.364415 18426 certs.go:437] found cert: /Users/jenkins/minikube-integration/17936-1021/.minikube/certs/Users/jenkins/minikube-integration/17936-1021/.minikube/certs/ca-key.pem (1679 bytes)
I0216 09:46:13.364445 18426 certs.go:437] found cert: /Users/jenkins/minikube-integration/17936-1021/.minikube/certs/Users/jenkins/minikube-integration/17936-1021/.minikube/certs/ca.pem (1082 bytes)
I0216 09:46:13.364476 18426 certs.go:437] found cert: /Users/jenkins/minikube-integration/17936-1021/.minikube/certs/Users/jenkins/minikube-integration/17936-1021/.minikube/certs/cert.pem (1123 bytes)
I0216 09:46:13.364511 18426 certs.go:437] found cert: /Users/jenkins/minikube-integration/17936-1021/.minikube/certs/Users/jenkins/minikube-integration/17936-1021/.minikube/certs/key.pem (1675 bytes)
I0216 09:46:13.364581 18426 certs.go:437] found cert: /Users/jenkins/minikube-integration/17936-1021/.minikube/files/etc/ssl/certs/Users/jenkins/minikube-integration/17936-1021/.minikube/files/etc/ssl/certs/21512.pem (1708 bytes)
I0216 09:46:13.365091 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1399 bytes)
I0216 09:46:13.406076 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1679 bytes)
I0216 09:46:13.448308 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0216 09:46:13.488101 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/profiles/old-k8s-version-356000/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1675 bytes)
I0216 09:46:13.530546 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0216 09:46:13.571162 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1675 bytes)
I0216 09:46:13.612992 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0216 09:46:13.652990 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1675 bytes)
I0216 09:46:13.693519 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/files/etc/ssl/certs/21512.pem --> /usr/share/ca-certificates/21512.pem (1708 bytes)
I0216 09:46:13.734264 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0216 09:46:13.774196 18426 ssh_runner.go:362] scp /Users/jenkins/minikube-integration/17936-1021/.minikube/certs/2151.pem --> /usr/share/ca-certificates/2151.pem (1338 bytes)
I0216 09:46:13.814625 18426 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0216 09:46:13.843583 18426 ssh_runner.go:195] Run: openssl version
I0216 09:46:13.850782 18426 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/21512.pem && ln -fs /usr/share/ca-certificates/21512.pem /etc/ssl/certs/21512.pem"
I0216 09:46:13.866387 18426 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/21512.pem
I0216 09:46:13.870825 18426 certs.go:480] hashing: -rw-r--r-- 1 root root 1708 Feb 16 16:51 /usr/share/ca-certificates/21512.pem
I0216 09:46:13.870883 18426 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/21512.pem
I0216 09:46:13.878124 18426 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/21512.pem /etc/ssl/certs/3ec20f2e.0"
I0216 09:46:13.894105 18426 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0216 09:46:13.910672 18426 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0216 09:46:13.915128 18426 certs.go:480] hashing: -rw-r--r-- 1 root root 1111 Feb 16 16:43 /usr/share/ca-certificates/minikubeCA.pem
I0216 09:46:13.915186 18426 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0216 09:46:13.922106 18426 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0216 09:46:13.939006 18426 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/2151.pem && ln -fs /usr/share/ca-certificates/2151.pem /etc/ssl/certs/2151.pem"
I0216 09:46:13.955137 18426 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/2151.pem
I0216 09:46:13.959595 18426 certs.go:480] hashing: -rw-r--r-- 1 root root 1338 Feb 16 16:51 /usr/share/ca-certificates/2151.pem
I0216 09:46:13.959647 18426 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/2151.pem
I0216 09:46:13.966606 18426 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/2151.pem /etc/ssl/certs/51391683.0"
I0216 09:46:13.982192 18426 ssh_runner.go:195] Run: ls /var/lib/minikube/certs/etcd
I0216 09:46:13.986433 18426 certs.go:353] certs directory doesn't exist, likely first start: ls /var/lib/minikube/certs/etcd: Process exited with status 2
stdout:
stderr:
ls: cannot access '/var/lib/minikube/certs/etcd': No such file or directory
I0216 09:46:13.986490 18426 kubeadm.go:404] StartCluster: {Name:old-k8s-version-356000 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-356000 Namespace:default APIServerName:minikubeCA APISer
verNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false Disab
leMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0216 09:46:13.986589 18426 ssh_runner.go:195] Run: docker ps --filter status=paused --filter=name=k8s_.*_(kube-system)_ --format={{.ID}}
I0216 09:46:14.006591 18426 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0216 09:46:14.021889 18426 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0216 09:46:14.037588 18426 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0216 09:46:14.037652 18426 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0216 09:46:14.052858 18426 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0216 09:46:14.052885 18426 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0216 09:46:14.111974 18426 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0216 09:46:14.112012 18426 kubeadm.go:322] [preflight] Running pre-flight checks
I0216 09:46:14.378386 18426 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0216 09:46:14.378506 18426 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0216 09:46:14.378610 18426 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0216 09:46:14.569100 18426 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0216 09:46:14.596824 18426 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0216 09:46:14.603457 18426 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0216 09:46:14.670318 18426 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0216 09:46:14.693002 18426 out.go:204] - Generating certificates and keys ...
I0216 09:46:14.693154 18426 kubeadm.go:322] [certs] Using existing ca certificate authority
I0216 09:46:14.693208 18426 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0216 09:46:14.954199 18426 kubeadm.go:322] [certs] Generating "apiserver-kubelet-client" certificate and key
I0216 09:46:15.203116 18426 kubeadm.go:322] [certs] Generating "front-proxy-ca" certificate and key
I0216 09:46:15.445856 18426 kubeadm.go:322] [certs] Generating "front-proxy-client" certificate and key
I0216 09:46:15.581305 18426 kubeadm.go:322] [certs] Generating "etcd/ca" certificate and key
I0216 09:46:15.641921 18426 kubeadm.go:322] [certs] Generating "etcd/server" certificate and key
I0216 09:46:15.642073 18426 kubeadm.go:322] [certs] etcd/server serving cert is signed for DNS names [old-k8s-version-356000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
I0216 09:46:15.751728 18426 kubeadm.go:322] [certs] Generating "etcd/peer" certificate and key
I0216 09:46:15.751890 18426 kubeadm.go:322] [certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-356000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
I0216 09:46:15.883425 18426 kubeadm.go:322] [certs] Generating "etcd/healthcheck-client" certificate and key
I0216 09:46:16.079878 18426 kubeadm.go:322] [certs] Generating "apiserver-etcd-client" certificate and key
I0216 09:46:16.245321 18426 kubeadm.go:322] [certs] Generating "sa" key and public key
I0216 09:46:16.245403 18426 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0216 09:46:16.356809 18426 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0216 09:46:16.550467 18426 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0216 09:46:16.688475 18426 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0216 09:46:16.819120 18426 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0216 09:46:16.820085 18426 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0216 09:46:16.846239 18426 out.go:204] - Booting up control plane ...
I0216 09:46:16.846332 18426 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0216 09:46:16.846423 18426 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0216 09:46:16.846510 18426 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0216 09:46:16.846601 18426 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0216 09:46:16.846784 18426 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0216 09:46:56.830617 18426 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0216 09:46:56.831037 18426 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0216 09:46:56.831215 18426 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0216 09:47:01.832175 18426 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0216 09:47:01.832408 18426 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0216 09:47:11.834080 18426 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0216 09:47:11.834255 18426 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0216 09:47:31.835663 18426 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0216 09:47:31.835813 18426 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0216 09:48:11.839484 18426 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0216 09:48:11.839714 18426 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0216 09:48:11.839733 18426 kubeadm.go:322]
I0216 09:48:11.839791 18426 kubeadm.go:322] Unfortunately, an error has occurred:
I0216 09:48:11.839839 18426 kubeadm.go:322] timed out waiting for the condition
I0216 09:48:11.839847 18426 kubeadm.go:322]
I0216 09:48:11.839911 18426 kubeadm.go:322] This error is likely caused by:
I0216 09:48:11.839943 18426 kubeadm.go:322] - The kubelet is not running
I0216 09:48:11.840044 18426 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0216 09:48:11.840055 18426 kubeadm.go:322]
I0216 09:48:11.840127 18426 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0216 09:48:11.840172 18426 kubeadm.go:322] - 'systemctl status kubelet'
I0216 09:48:11.840203 18426 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0216 09:48:11.840210 18426 kubeadm.go:322]
I0216 09:48:11.840299 18426 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0216 09:48:11.840374 18426 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0216 09:48:11.840440 18426 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0216 09:48:11.840480 18426 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0216 09:48:11.840544 18426 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0216 09:48:11.840570 18426 kubeadm.go:322] - 'docker logs CONTAINERID'
I0216 09:48:11.844675 18426 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0216 09:48:11.844758 18426 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0216 09:48:11.844858 18426 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
I0216 09:48:11.844962 18426 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0216 09:48:11.845031 18426 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0216 09:48:11.845093 18426 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
W0216 09:48:11.845172 18426 out.go:239] ! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-356000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-356000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-356000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-356000 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
I0216 09:48:11.845216 18426 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm reset --cri-socket /var/run/dockershim.sock --force"
I0216 09:48:12.262174 18426 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service kubelet
I0216 09:48:12.279300 18426 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0216 09:48:12.279361 18426 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0216 09:48:12.294464 18426 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0216 09:48:12.294500 18426 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0216 09:48:12.347101 18426 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0216 09:48:12.347141 18426 kubeadm.go:322] [preflight] Running pre-flight checks
I0216 09:48:12.574546 18426 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0216 09:48:12.574658 18426 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0216 09:48:12.574750 18426 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0216 09:48:12.729920 18426 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0216 09:48:12.730651 18426 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0216 09:48:12.737053 18426 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0216 09:48:12.799801 18426 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0216 09:48:12.821323 18426 out.go:204] - Generating certificates and keys ...
I0216 09:48:12.821454 18426 kubeadm.go:322] [certs] Using existing ca certificate authority
I0216 09:48:12.821525 18426 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0216 09:48:12.821621 18426 kubeadm.go:322] [certs] Using existing apiserver-kubelet-client certificate and key on disk
I0216 09:48:12.821688 18426 kubeadm.go:322] [certs] Using existing front-proxy-ca certificate authority
I0216 09:48:12.821780 18426 kubeadm.go:322] [certs] Using existing front-proxy-client certificate and key on disk
I0216 09:48:12.821838 18426 kubeadm.go:322] [certs] Using existing etcd/ca certificate authority
I0216 09:48:12.821906 18426 kubeadm.go:322] [certs] Using existing etcd/server certificate and key on disk
I0216 09:48:12.821979 18426 kubeadm.go:322] [certs] Using existing etcd/peer certificate and key on disk
I0216 09:48:12.822037 18426 kubeadm.go:322] [certs] Using existing etcd/healthcheck-client certificate and key on disk
I0216 09:48:12.822157 18426 kubeadm.go:322] [certs] Using existing apiserver-etcd-client certificate and key on disk
I0216 09:48:12.822258 18426 kubeadm.go:322] [certs] Using the existing "sa" key
I0216 09:48:12.822352 18426 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0216 09:48:13.017676 18426 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0216 09:48:13.238901 18426 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0216 09:48:13.552141 18426 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0216 09:48:13.592668 18426 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0216 09:48:13.593346 18426 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0216 09:48:13.614911 18426 out.go:204] - Booting up control plane ...
I0216 09:48:13.615008 18426 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0216 09:48:13.615096 18426 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0216 09:48:13.615185 18426 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0216 09:48:13.615270 18426 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0216 09:48:13.615439 18426 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0216 09:48:53.603282 18426 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0216 09:48:53.603593 18426 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0216 09:48:53.603799 18426 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0216 09:48:58.605252 18426 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0216 09:48:58.605392 18426 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0216 09:49:08.606399 18426 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0216 09:49:08.606556 18426 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0216 09:49:28.608224 18426 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0216 09:49:28.608373 18426 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0216 09:50:08.611752 18426 kubeadm.go:322] [kubelet-check] It seems like the kubelet isn't running or healthy.
I0216 09:50:08.611907 18426 kubeadm.go:322] [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
I0216 09:50:08.611915 18426 kubeadm.go:322]
I0216 09:50:08.611944 18426 kubeadm.go:322] Unfortunately, an error has occurred:
I0216 09:50:08.611983 18426 kubeadm.go:322] timed out waiting for the condition
I0216 09:50:08.611994 18426 kubeadm.go:322]
I0216 09:50:08.612020 18426 kubeadm.go:322] This error is likely caused by:
I0216 09:50:08.612049 18426 kubeadm.go:322] - The kubelet is not running
I0216 09:50:08.612141 18426 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0216 09:50:08.612153 18426 kubeadm.go:322]
I0216 09:50:08.612230 18426 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0216 09:50:08.612257 18426 kubeadm.go:322] - 'systemctl status kubelet'
I0216 09:50:08.612279 18426 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0216 09:50:08.612285 18426 kubeadm.go:322]
I0216 09:50:08.612364 18426 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0216 09:50:08.612449 18426 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0216 09:50:08.612522 18426 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0216 09:50:08.612564 18426 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0216 09:50:08.612628 18426 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0216 09:50:08.612661 18426 kubeadm.go:322] - 'docker logs CONTAINERID'
I0216 09:50:08.616485 18426 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0216 09:50:08.616546 18426 kubeadm.go:322] [WARNING Swap]: running with swap on is not supported. Please disable swap
I0216 09:50:08.616646 18426 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
I0216 09:50:08.616729 18426 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0216 09:50:08.616800 18426 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0216 09:50:08.616856 18426 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
I0216 09:50:08.616904 18426 kubeadm.go:406] StartCluster complete in 3m54.625836035s
I0216 09:50:08.616987 18426 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-apiserver --format={{.ID}}
I0216 09:50:08.633359 18426 logs.go:276] 0 containers: []
W0216 09:50:08.633372 18426 logs.go:278] No container was found matching "kube-apiserver"
I0216 09:50:08.633447 18426 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_etcd --format={{.ID}}
I0216 09:50:08.652092 18426 logs.go:276] 0 containers: []
W0216 09:50:08.652106 18426 logs.go:278] No container was found matching "etcd"
I0216 09:50:08.652175 18426 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_coredns --format={{.ID}}
I0216 09:50:08.669245 18426 logs.go:276] 0 containers: []
W0216 09:50:08.669259 18426 logs.go:278] No container was found matching "coredns"
I0216 09:50:08.669332 18426 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-scheduler --format={{.ID}}
I0216 09:50:08.686142 18426 logs.go:276] 0 containers: []
W0216 09:50:08.686158 18426 logs.go:278] No container was found matching "kube-scheduler"
I0216 09:50:08.686226 18426 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-proxy --format={{.ID}}
I0216 09:50:08.703601 18426 logs.go:276] 0 containers: []
W0216 09:50:08.703615 18426 logs.go:278] No container was found matching "kube-proxy"
I0216 09:50:08.703685 18426 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-controller-manager --format={{.ID}}
I0216 09:50:08.720435 18426 logs.go:276] 0 containers: []
W0216 09:50:08.720486 18426 logs.go:278] No container was found matching "kube-controller-manager"
I0216 09:50:08.720603 18426 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kindnet --format={{.ID}}
I0216 09:50:08.737690 18426 logs.go:276] 0 containers: []
W0216 09:50:08.737704 18426 logs.go:278] No container was found matching "kindnet"
I0216 09:50:08.737712 18426 logs.go:123] Gathering logs for kubelet ...
I0216 09:50:08.737720 18426 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
I0216 09:50:08.784335 18426 logs.go:123] Gathering logs for dmesg ...
I0216 09:50:08.784355 18426 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0216 09:50:08.820035 18426 logs.go:123] Gathering logs for describe nodes ...
I0216 09:50:08.820053 18426 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0216 09:50:08.907213 18426 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0216 09:50:08.907227 18426 logs.go:123] Gathering logs for Docker ...
I0216 09:50:08.907241 18426 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u docker -u cri-docker -n 400"
I0216 09:50:08.929671 18426 logs.go:123] Gathering logs for container status ...
I0216 09:50:08.929686 18426 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
W0216 09:50:08.991803 18426 out.go:364] Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0216 09:50:08.991827 18426 out.go:239] *
*
W0216 09:50:08.991867 18426 out.go:239] X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0216 09:50:08.991900 18426 out.go:239] *
*
W0216 09:50:08.992531 18426 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0216 09:50:09.056346 18426 out.go:177]
W0216 09:50:09.099071 18426 out.go:239] X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0216 09:50:09.099120 18426 out.go:239] * Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
* Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
W0216 09:50:09.099140 18426 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/4172
* Related issue: https://github.com/kubernetes/minikube/issues/4172
I0216 09:50:09.120236 18426 out.go:177]
** /stderr **
start_stop_delete_test.go:188: failed starting minikube -first start-. args "out/minikube-darwin-amd64 start -p old-k8s-version-356000 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --kubernetes-version=v1.16.0": exit status 109
helpers_test.go:222: -----------------------post-mortem--------------------------------
helpers_test.go:230: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/FirstStart]: docker inspect <======
helpers_test.go:231: (dbg) Run: docker inspect old-k8s-version-356000
helpers_test.go:235: (dbg) docker inspect old-k8s-version-356000:
-- stdout --
[
{
"Id": "c7e40ba5a933a3fc62b7591c10e5e9c4a9dacef26179c3d4d877828d0eb3ca01",
"Created": "2024-02-16T17:45:56.532939996Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 350253,
"ExitCode": 0,
"Error": "",
"StartedAt": "2024-02-16T17:45:56.786241358Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:78bbddd92c7656e5a7bf9651f59e6b47aa97241efd2e93247c457ec76b2185c5",
"ResolvConfPath": "/var/lib/docker/containers/c7e40ba5a933a3fc62b7591c10e5e9c4a9dacef26179c3d4d877828d0eb3ca01/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/c7e40ba5a933a3fc62b7591c10e5e9c4a9dacef26179c3d4d877828d0eb3ca01/hostname",
"HostsPath": "/var/lib/docker/containers/c7e40ba5a933a3fc62b7591c10e5e9c4a9dacef26179c3d4d877828d0eb3ca01/hosts",
"LogPath": "/var/lib/docker/containers/c7e40ba5a933a3fc62b7591c10e5e9c4a9dacef26179c3d4d877828d0eb3ca01/c7e40ba5a933a3fc62b7591c10e5e9c4a9dacef26179c3d4d877828d0eb3ca01-json.log",
"Name": "/old-k8s-version-356000",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro",
"old-k8s-version-356000:/var"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "old-k8s-version-356000",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "private",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 2306867200,
"NanoCpus": 2000000000,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": [],
"BlkioDeviceWriteBps": [],
"BlkioDeviceReadIOps": [],
"BlkioDeviceWriteIOps": [],
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 2306867200,
"MemorySwappiness": null,
"OomKillDisable": null,
"PidsLimit": null,
"Ulimits": [],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/2a00a90b2099718abfd0fc15d9a576cacf4ce76139e48d7d0a0d8eaa01bf4379-init/diff:/var/lib/docker/overlay2/64e9a96b4fa04416cc2f23ab4bb4beb68546d8c810a8f2c9b8ab796aea7581a7/diff",
"MergedDir": "/var/lib/docker/overlay2/2a00a90b2099718abfd0fc15d9a576cacf4ce76139e48d7d0a0d8eaa01bf4379/merged",
"UpperDir": "/var/lib/docker/overlay2/2a00a90b2099718abfd0fc15d9a576cacf4ce76139e48d7d0a0d8eaa01bf4379/diff",
"WorkDir": "/var/lib/docker/overlay2/2a00a90b2099718abfd0fc15d9a576cacf4ce76139e48d7d0a0d8eaa01bf4379/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "old-k8s-version-356000",
"Source": "/var/lib/docker/volumes/old-k8s-version-356000/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "old-k8s-version-356000",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"22/tcp": {},
"2376/tcp": {},
"32443/tcp": {},
"5000/tcp": {},
"8443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "old-k8s-version-356000",
"name.minikube.sigs.k8s.io": "old-k8s-version-356000",
"role.minikube.sigs.k8s.io": ""
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "316a645e6070a04df20f8a3c2b5778a2943dd3ff2014cf5c036ba10fc84a1550",
"SandboxKey": "/var/run/docker/netns/316a645e6070",
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "53808"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "53809"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "53810"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "53806"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "53807"
}
]
},
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"old-k8s-version-356000": {
"IPAMConfig": {
"IPv4Address": "192.168.76.2"
},
"Links": null,
"Aliases": [
"c7e40ba5a933",
"old-k8s-version-356000"
],
"MacAddress": "02:42:c0:a8:4c:02",
"NetworkID": "2b231f9382e31cc79f696866baa9c7eea268e7a10c9edda380cefa5e7ba22d21",
"EndpointID": "bdb22bb2035679a82685e03e38fabe1078df25fe59cde9a2fd156de399fe1054",
"Gateway": "192.168.76.1",
"IPAddress": "192.168.76.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"DriverOpts": null,
"DNSNames": [
"old-k8s-version-356000",
"c7e40ba5a933"
]
}
}
}
}
]
-- /stdout --
helpers_test.go:239: (dbg) Run: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-356000 -n old-k8s-version-356000
helpers_test.go:239: (dbg) Non-zero exit: out/minikube-darwin-amd64 status --format={{.Host}} -p old-k8s-version-356000 -n old-k8s-version-356000: exit status 6 (435.127395ms)
-- stdout --
Running
WARNING: Your kubectl is pointing to stale minikube-vm.
To fix the kubectl context, run `minikube update-context`
-- /stdout --
** stderr **
E0216 09:50:09.694705 19295 status.go:415] kubeconfig endpoint: extract IP: "old-k8s-version-356000" does not appear in /Users/jenkins/minikube-integration/17936-1021/kubeconfig
** /stderr **
helpers_test.go:239: status error: exit status 6 (may be ok)
helpers_test.go:241: "old-k8s-version-356000" host is not running, skipping log retrieval (state="Running\nWARNING: Your kubectl is pointing to stale minikube-vm.\nTo fix the kubectl context, run `minikube update-context`")
--- FAIL: TestStartStop/group/old-k8s-version/serial/FirstStart (257.46s)