=== RUN TestStartStop/group/old-k8s-version/serial/FirstStart
start_stop_delete_test.go:186: (dbg) Run: out/minikube-linux-amd64 start -p old-k8s-version-799707 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --container-runtime=docker --kubernetes-version=v1.16.0
start_stop_delete_test.go:186: (dbg) Non-zero exit: out/minikube-linux-amd64 start -p old-k8s-version-799707 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --container-runtime=docker --kubernetes-version=v1.16.0: exit status 109 (8m26.54539692s)
-- stdout --
* [old-k8s-version-799707] minikube v1.32.0 on Ubuntu 20.04 (kvm/amd64)
- MINIKUBE_LOCATION=18233
- MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
- KUBECONFIG=/home/jenkins/minikube-integration/18233-317564/kubeconfig
- MINIKUBE_HOME=/home/jenkins/minikube-integration/18233-317564/.minikube
- MINIKUBE_BIN=out/minikube-linux-amd64
- MINIKUBE_FORCE_SYSTEMD=
* Using the docker driver based on user configuration
* Using Docker driver with root privileges
* Starting control plane node old-k8s-version-799707 in cluster old-k8s-version-799707
* Pulling base image v0.0.42-1708008208-17936 ...
* Creating docker container (CPUs=2, Memory=2200MB) ...
* Preparing Kubernetes v1.16.0 on Docker 25.0.3 ...
- Generating certificates and keys ...
- Booting up control plane ...
- Generating certificates and keys ...
- Booting up control plane ...
X Problems detected in kubelet:
Feb 23 01:23:02 old-k8s-version-799707 kubelet[5695]: E0223 01:23:02.555197 5695 pod_workers.go:191] Error syncing pod b3d303074fe0ca1d42a8bd9ed248df09 ("kube-scheduler-old-k8s-version-799707_kube-system(b3d303074fe0ca1d42a8bd9ed248df09)"), skipping: failed to "StartContainer" for "kube-scheduler" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-scheduler:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-scheduler:v1.16.0\" is not set"
Feb 23 01:23:05 old-k8s-version-799707 kubelet[5695]: E0223 01:23:05.555141 5695 pod_workers.go:191] Error syncing pod 9fc427d2e6746d2b3f18846f6f0fcafb ("kube-controller-manager-old-k8s-version-799707_kube-system(9fc427d2e6746d2b3f18846f6f0fcafb)"), skipping: failed to "StartContainer" for "kube-controller-manager" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-controller-manager:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-controller-manager:v1.16.0\" is not set"
Feb 23 01:23:06 old-k8s-version-799707 kubelet[5695]: E0223 01:23:06.555495 5695 pod_workers.go:191] Error syncing pod ada9a48d69d0428e9c15f5ad9c81ef07 ("etcd-old-k8s-version-799707_kube-system(ada9a48d69d0428e9c15f5ad9c81ef07)"), skipping: failed to "StartContainer" for "etcd" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/etcd:3.3.15-0\": Id or size of image \"k8s.gcr.io/etcd:3.3.15-0\" is not set"
-- /stdout --
** stderr **
I0223 01:14:56.753006 662830 out.go:291] Setting OutFile to fd 1 ...
I0223 01:14:56.753183 662830 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0223 01:14:56.753194 662830 out.go:304] Setting ErrFile to fd 2...
I0223 01:14:56.753201 662830 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0223 01:14:56.753482 662830 root.go:338] Updating PATH: /home/jenkins/minikube-integration/18233-317564/.minikube/bin
I0223 01:14:56.754220 662830 out.go:298] Setting JSON to false
I0223 01:14:56.755677 662830 start.go:129] hostinfo: {"hostname":"ubuntu-20-agent-15","uptime":7046,"bootTime":1708643851,"procs":459,"os":"linux","platform":"ubuntu","platformFamily":"debian","platformVersion":"20.04","kernelVersion":"5.15.0-1051-gcp","kernelArch":"x86_64","virtualizationSystem":"kvm","virtualizationRole":"guest","hostId":"591c9f12-2938-3743-e2bf-c56a050d43d1"}
I0223 01:14:56.755759 662830 start.go:139] virtualization: kvm guest
I0223 01:14:56.758083 662830 out.go:177] * [old-k8s-version-799707] minikube v1.32.0 on Ubuntu 20.04 (kvm/amd64)
I0223 01:14:56.759729 662830 out.go:177] - MINIKUBE_LOCATION=18233
I0223 01:14:56.759750 662830 notify.go:220] Checking for updates...
I0223 01:14:56.762376 662830 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0223 01:14:56.763831 662830 out.go:177] - KUBECONFIG=/home/jenkins/minikube-integration/18233-317564/kubeconfig
I0223 01:14:56.765364 662830 out.go:177] - MINIKUBE_HOME=/home/jenkins/minikube-integration/18233-317564/.minikube
I0223 01:14:56.766768 662830 out.go:177] - MINIKUBE_BIN=out/minikube-linux-amd64
I0223 01:14:56.768279 662830 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0223 01:14:56.770309 662830 config.go:182] Loaded profile config "enable-default-cni-600346": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.28.4
I0223 01:14:56.770481 662830 config.go:182] Loaded profile config "kubenet-600346": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.28.4
I0223 01:14:56.770597 662830 config.go:182] Loaded profile config "kubernetes-upgrade-849442": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.16.0
I0223 01:14:56.770728 662830 driver.go:392] Setting default libvirt URI to qemu:///system
I0223 01:14:56.799470 662830 docker.go:122] docker version: linux-25.0.3:Docker Engine - Community
I0223 01:14:56.799618 662830 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0223 01:14:56.862794 662830 info.go:266] docker info: {ID:TS6T:UINC:MIYS:RZPA:KS6T:4JQK:7JHN:D6RA:LDP2:MHAE:G32M:C5NQ Containers:2 ContainersRunning:2 ContainersPaused:0 ContainersStopped:0 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:53 OomKillDisable:true NGoroutines:80 SystemTime:2024-02-23 01:14:56.850889026 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1051-gcp OperatingSystem:Ubuntu 20.04.6 LTS OSType:linux Architecture:x86
_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:8 MemTotal:33647996928 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ubuntu-20-agent-15 Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=builtin] ProductLicense: Warnings:<nil> ServerEr
rors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.5] map[Name:scan Path:/usr/libexec/docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.23.0]] Warnings:<nil>}}
I0223 01:14:56.862902 662830 docker.go:295] overlay module found
I0223 01:14:56.864874 662830 out.go:177] * Using the docker driver based on user configuration
I0223 01:14:56.866226 662830 start.go:299] selected driver: docker
I0223 01:14:56.866242 662830 start.go:903] validating driver "docker" against <nil>
I0223 01:14:56.866254 662830 start.go:914] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0223 01:14:56.867187 662830 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0223 01:14:56.933848 662830 info.go:266] docker info: {ID:TS6T:UINC:MIYS:RZPA:KS6T:4JQK:7JHN:D6RA:LDP2:MHAE:G32M:C5NQ Containers:2 ContainersRunning:2 ContainersPaused:0 ContainersStopped:0 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:51 OomKillDisable:true NGoroutines:77 SystemTime:2024-02-23 01:14:56.92293351 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1051-gcp OperatingSystem:Ubuntu 20.04.6 LTS OSType:linux Architecture:x86_
64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:8 MemTotal:33647996928 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ubuntu-20-agent-15 Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=builtin] ProductLicense: Warnings:<nil> ServerErr
ors:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.5] map[Name:scan Path:/usr/libexec/docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.23.0]] Warnings:<nil>}}
I0223 01:14:56.934026 662830 start_flags.go:309] no existing cluster config was found, will generate one from the flags
I0223 01:14:56.934708 662830 start_flags.go:931] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0223 01:14:56.936509 662830 out.go:177] * Using Docker driver with root privileges
I0223 01:14:56.937941 662830 cni.go:84] Creating CNI manager for ""
I0223 01:14:56.937973 662830 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0223 01:14:56.937988 662830 start_flags.go:323] config:
{Name:old-k8s-version-799707 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-799707 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local Containe
rRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0223 01:14:56.939764 662830 out.go:177] * Starting control plane node old-k8s-version-799707 in cluster old-k8s-version-799707
I0223 01:14:56.941145 662830 cache.go:121] Beginning downloading kic base image for docker with docker
I0223 01:14:56.943284 662830 out.go:177] * Pulling base image v0.0.42-1708008208-17936 ...
I0223 01:14:56.944718 662830 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0223 01:14:56.944763 662830 preload.go:148] Found local preload: /home/jenkins/minikube-integration/18233-317564/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4
I0223 01:14:56.944779 662830 cache.go:56] Caching tarball of preloaded images
I0223 01:14:56.944783 662830 image.go:79] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf in local docker daemon
I0223 01:14:56.944855 662830 preload.go:174] Found /home/jenkins/minikube-integration/18233-317564/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0223 01:14:56.944868 662830 cache.go:59] Finished verifying existence of preloaded tar for v1.16.0 on docker
I0223 01:14:56.945000 662830 profile.go:148] Saving config to /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/config.json ...
I0223 01:14:56.945022 662830 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/config.json: {Name:mkd1d8c2e3454dfcc4d18803053dd224c3f53d4d Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0223 01:14:56.963276 662830 image.go:83] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf in local docker daemon, skipping pull
I0223 01:14:56.963309 662830 cache.go:144] gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf exists in daemon, skipping load
I0223 01:14:56.963337 662830 cache.go:194] Successfully downloaded all kic artifacts
I0223 01:14:56.963382 662830 start.go:365] acquiring machines lock for old-k8s-version-799707: {Name:mkec58acc477a1259ea890fef71c8d064abcdc6e Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0223 01:14:56.963507 662830 start.go:369] acquired machines lock for "old-k8s-version-799707" in 100.39µs
I0223 01:14:56.963545 662830 start.go:93] Provisioning new machine with config: &{Name:old-k8s-version-799707 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-799707 Namespace:default APIServer
Name:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimiza
tions:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s} &{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}
I0223 01:14:56.963697 662830 start.go:125] createHost starting for "" (driver="docker")
I0223 01:14:56.969729 662830 out.go:204] * Creating docker container (CPUs=2, Memory=2200MB) ...
I0223 01:14:56.970045 662830 start.go:159] libmachine.API.Create for "old-k8s-version-799707" (driver="docker")
I0223 01:14:56.970101 662830 client.go:168] LocalClient.Create starting
I0223 01:14:56.970198 662830 main.go:141] libmachine: Reading certificate data from /home/jenkins/minikube-integration/18233-317564/.minikube/certs/ca.pem
I0223 01:14:56.970239 662830 main.go:141] libmachine: Decoding PEM data...
I0223 01:14:56.970256 662830 main.go:141] libmachine: Parsing certificate...
I0223 01:14:56.970319 662830 main.go:141] libmachine: Reading certificate data from /home/jenkins/minikube-integration/18233-317564/.minikube/certs/cert.pem
I0223 01:14:56.970352 662830 main.go:141] libmachine: Decoding PEM data...
I0223 01:14:56.970370 662830 main.go:141] libmachine: Parsing certificate...
I0223 01:14:56.970742 662830 cli_runner.go:164] Run: docker network inspect old-k8s-version-799707 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
W0223 01:14:56.999086 662830 cli_runner.go:211] docker network inspect old-k8s-version-799707 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1
I0223 01:14:56.999197 662830 network_create.go:281] running [docker network inspect old-k8s-version-799707] to gather additional debugging logs...
I0223 01:14:56.999225 662830 cli_runner.go:164] Run: docker network inspect old-k8s-version-799707
W0223 01:14:57.020090 662830 cli_runner.go:211] docker network inspect old-k8s-version-799707 returned with exit code 1
I0223 01:14:57.020124 662830 network_create.go:284] error running [docker network inspect old-k8s-version-799707]: docker network inspect old-k8s-version-799707: exit status 1
stdout:
[]
stderr:
Error response from daemon: network old-k8s-version-799707 not found
I0223 01:14:57.020141 662830 network_create.go:286] output of [docker network inspect old-k8s-version-799707]: -- stdout --
[]
-- /stdout --
** stderr **
Error response from daemon: network old-k8s-version-799707 not found
** /stderr **
I0223 01:14:57.020273 662830 cli_runner.go:164] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0223 01:14:57.038021 662830 network.go:212] skipping subnet 192.168.49.0/24 that is taken: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName:br-695ca2766a58 IfaceIPv4:192.168.49.1 IfaceMTU:1500 IfaceMAC:02:42:f4:90:90:f1} reservation:<nil>}
I0223 01:14:57.038848 662830 network.go:212] skipping subnet 192.168.58.0/24 that is taken: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 IsPrivate:true Interface:{IfaceName:br-db2da4de8123 IfaceIPv4:192.168.58.1 IfaceMTU:1500 IfaceMAC:02:42:bf:69:28:2c} reservation:<nil>}
I0223 01:14:57.039484 662830 network.go:212] skipping subnet 192.168.67.0/24 that is taken: &{IP:192.168.67.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.67.0/24 Gateway:192.168.67.1 ClientMin:192.168.67.2 ClientMax:192.168.67.254 Broadcast:192.168.67.255 IsPrivate:true Interface:{IfaceName:br-8049eee158fa IfaceIPv4:192.168.67.1 IfaceMTU:1500 IfaceMAC:02:42:10:7e:b5:e3} reservation:<nil>}
I0223 01:14:57.040023 662830 network.go:212] skipping subnet 192.168.76.0/24 that is taken: &{IP:192.168.76.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.76.0/24 Gateway:192.168.76.1 ClientMin:192.168.76.2 ClientMax:192.168.76.254 Broadcast:192.168.76.255 IsPrivate:true Interface:{IfaceName:br-fddd82e2b023 IfaceIPv4:192.168.76.1 IfaceMTU:1500 IfaceMAC:02:42:5e:7b:0e:60} reservation:<nil>}
I0223 01:14:57.040639 662830 network.go:212] skipping subnet 192.168.85.0/24 that is taken: &{IP:192.168.85.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.85.0/24 Gateway:192.168.85.1 ClientMin:192.168.85.2 ClientMax:192.168.85.254 Broadcast:192.168.85.255 IsPrivate:true Interface:{IfaceName:br-95ad12b8a760 IfaceIPv4:192.168.85.1 IfaceMTU:1500 IfaceMAC:02:42:1e:b8:26:be} reservation:<nil>}
I0223 01:14:57.041267 662830 network.go:207] using free private subnet 192.168.94.0/24: &{IP:192.168.94.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.94.0/24 Gateway:192.168.94.1 ClientMin:192.168.94.2 ClientMax:192.168.94.254 Broadcast:192.168.94.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc002245a10}
I0223 01:14:57.041288 662830 network_create.go:124] attempt to create docker network old-k8s-version-799707 192.168.94.0/24 with gateway 192.168.94.1 and MTU of 1500 ...
I0223 01:14:57.041330 662830 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.94.0/24 --gateway=192.168.94.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=1500 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-799707 old-k8s-version-799707
I0223 01:14:57.107392 662830 network_create.go:108] docker network old-k8s-version-799707 192.168.94.0/24 created
I0223 01:14:57.107442 662830 kic.go:121] calculated static IP "192.168.94.2" for the "old-k8s-version-799707" container
I0223 01:14:57.107539 662830 cli_runner.go:164] Run: docker ps -a --format {{.Names}}
I0223 01:14:57.129921 662830 cli_runner.go:164] Run: docker volume create old-k8s-version-799707 --label name.minikube.sigs.k8s.io=old-k8s-version-799707 --label created_by.minikube.sigs.k8s.io=true
I0223 01:14:57.157191 662830 oci.go:103] Successfully created a docker volume old-k8s-version-799707
I0223 01:14:57.157294 662830 cli_runner.go:164] Run: docker run --rm --name old-k8s-version-799707-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-799707 --entrypoint /usr/bin/test -v old-k8s-version-799707:/var gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf -d /var/lib
I0223 01:14:57.836239 662830 oci.go:107] Successfully prepared a docker volume old-k8s-version-799707
I0223 01:14:57.836285 662830 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0223 01:14:57.836309 662830 kic.go:194] Starting extracting preloaded images to volume ...
I0223 01:14:57.836372 662830 cli_runner.go:164] Run: docker run --rm --entrypoint /usr/bin/tar -v /home/jenkins/minikube-integration/18233-317564/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-799707:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf -I lz4 -xf /preloaded.tar -C /extractDir
I0223 01:15:05.391881 662830 cli_runner.go:217] Completed: docker run --rm --entrypoint /usr/bin/tar -v /home/jenkins/minikube-integration/18233-317564/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-799707:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf -I lz4 -xf /preloaded.tar -C /extractDir: (7.555440631s)
I0223 01:15:05.391922 662830 kic.go:203] duration metric: took 7.555609 seconds to extract preloaded images to volume
W0223 01:15:05.392070 662830 cgroups_linux.go:77] Your kernel does not support swap limit capabilities or the cgroup is not mounted.
I0223 01:15:05.392184 662830 cli_runner.go:164] Run: docker info --format "'{{json .SecurityOptions}}'"
I0223 01:15:05.456562 662830 cli_runner.go:164] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname old-k8s-version-799707 --name old-k8s-version-799707 --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-799707 --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=old-k8s-version-799707 --network old-k8s-version-799707 --ip 192.168.94.2 --volume old-k8s-version-799707:/var --security-opt apparmor=unconfined --memory=2200mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf
I0223 01:15:05.806732 662830 cli_runner.go:164] Run: docker container inspect old-k8s-version-799707 --format={{.State.Running}}
I0223 01:15:05.827061 662830 cli_runner.go:164] Run: docker container inspect old-k8s-version-799707 --format={{.State.Status}}
I0223 01:15:05.847132 662830 cli_runner.go:164] Run: docker exec old-k8s-version-799707 stat /var/lib/dpkg/alternatives/iptables
I0223 01:15:05.897318 662830 oci.go:144] the created container "old-k8s-version-799707" has a running status.
I0223 01:15:05.897356 662830 kic.go:225] Creating ssh key for kic: /home/jenkins/minikube-integration/18233-317564/.minikube/machines/old-k8s-version-799707/id_rsa...
I0223 01:15:06.326861 662830 kic_runner.go:191] docker (temp): /home/jenkins/minikube-integration/18233-317564/.minikube/machines/old-k8s-version-799707/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0223 01:15:06.354546 662830 cli_runner.go:164] Run: docker container inspect old-k8s-version-799707 --format={{.State.Status}}
I0223 01:15:06.395841 662830 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0223 01:15:06.395866 662830 kic_runner.go:114] Args: [docker exec --privileged old-k8s-version-799707 chown docker:docker /home/docker/.ssh/authorized_keys]
I0223 01:15:06.492024 662830 cli_runner.go:164] Run: docker container inspect old-k8s-version-799707 --format={{.State.Status}}
I0223 01:15:06.518150 662830 machine.go:88] provisioning docker machine ...
I0223 01:15:06.518192 662830 ubuntu.go:169] provisioning hostname "old-k8s-version-799707"
I0223 01:15:06.518262 662830 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-799707
I0223 01:15:06.544145 662830 main.go:141] libmachine: Using SSH client type: native
I0223 01:15:06.544454 662830 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x82d280] 0x82ffe0 <nil> [] 0s} 127.0.0.1 33358 <nil> <nil>}
I0223 01:15:06.544478 662830 main.go:141] libmachine: About to run SSH command:
sudo hostname old-k8s-version-799707 && echo "old-k8s-version-799707" | sudo tee /etc/hostname
I0223 01:15:06.701364 662830 main.go:141] libmachine: SSH cmd err, output: <nil>: old-k8s-version-799707
I0223 01:15:06.701469 662830 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-799707
I0223 01:15:06.720494 662830 main.go:141] libmachine: Using SSH client type: native
I0223 01:15:06.720714 662830 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x82d280] 0x82ffe0 <nil> [] 0s} 127.0.0.1 33358 <nil> <nil>}
I0223 01:15:06.720743 662830 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sold-k8s-version-799707' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 old-k8s-version-799707/g' /etc/hosts;
else
echo '127.0.1.1 old-k8s-version-799707' | sudo tee -a /etc/hosts;
fi
fi
I0223 01:15:06.857809 662830 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0223 01:15:06.857843 662830 ubuntu.go:175] set auth options {CertDir:/home/jenkins/minikube-integration/18233-317564/.minikube CaCertPath:/home/jenkins/minikube-integration/18233-317564/.minikube/certs/ca.pem CaPrivateKeyPath:/home/jenkins/minikube-integration/18233-317564/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/home/jenkins/minikube-integration/18233-317564/.minikube/machines/server.pem ServerKeyPath:/home/jenkins/minikube-integration/18233-317564/.minikube/machines/server-key.pem ClientKeyPath:/home/jenkins/minikube-integration/18233-317564/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/home/jenkins/minikube-integration/18233-317564/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/home/jenkins/minikube-integration/18233-317564/.minikube}
I0223 01:15:06.857893 662830 ubuntu.go:177] setting up certificates
I0223 01:15:06.857911 662830 provision.go:83] configureAuth start
I0223 01:15:06.857973 662830 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-799707
I0223 01:15:06.874371 662830 provision.go:138] copyHostCerts
I0223 01:15:06.874427 662830 exec_runner.go:144] found /home/jenkins/minikube-integration/18233-317564/.minikube/key.pem, removing ...
I0223 01:15:06.874436 662830 exec_runner.go:203] rm: /home/jenkins/minikube-integration/18233-317564/.minikube/key.pem
I0223 01:15:06.874498 662830 exec_runner.go:151] cp: /home/jenkins/minikube-integration/18233-317564/.minikube/certs/key.pem --> /home/jenkins/minikube-integration/18233-317564/.minikube/key.pem (1675 bytes)
I0223 01:15:06.874588 662830 exec_runner.go:144] found /home/jenkins/minikube-integration/18233-317564/.minikube/ca.pem, removing ...
I0223 01:15:06.874599 662830 exec_runner.go:203] rm: /home/jenkins/minikube-integration/18233-317564/.minikube/ca.pem
I0223 01:15:06.874622 662830 exec_runner.go:151] cp: /home/jenkins/minikube-integration/18233-317564/.minikube/certs/ca.pem --> /home/jenkins/minikube-integration/18233-317564/.minikube/ca.pem (1078 bytes)
I0223 01:15:06.874706 662830 exec_runner.go:144] found /home/jenkins/minikube-integration/18233-317564/.minikube/cert.pem, removing ...
I0223 01:15:06.874717 662830 exec_runner.go:203] rm: /home/jenkins/minikube-integration/18233-317564/.minikube/cert.pem
I0223 01:15:06.874748 662830 exec_runner.go:151] cp: /home/jenkins/minikube-integration/18233-317564/.minikube/certs/cert.pem --> /home/jenkins/minikube-integration/18233-317564/.minikube/cert.pem (1123 bytes)
I0223 01:15:06.874812 662830 provision.go:112] generating server cert: /home/jenkins/minikube-integration/18233-317564/.minikube/machines/server.pem ca-key=/home/jenkins/minikube-integration/18233-317564/.minikube/certs/ca.pem private-key=/home/jenkins/minikube-integration/18233-317564/.minikube/certs/ca-key.pem org=jenkins.old-k8s-version-799707 san=[192.168.94.2 127.0.0.1 localhost 127.0.0.1 minikube old-k8s-version-799707]
I0223 01:15:06.953695 662830 provision.go:172] copyRemoteCerts
I0223 01:15:06.953770 662830 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0223 01:15:06.953819 662830 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-799707
I0223 01:15:06.975670 662830 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33358 SSHKeyPath:/home/jenkins/minikube-integration/18233-317564/.minikube/machines/old-k8s-version-799707/id_rsa Username:docker}
I0223 01:15:07.075894 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1078 bytes)
I0223 01:15:07.100902 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/machines/server.pem --> /etc/docker/server.pem (1241 bytes)
I0223 01:15:07.123753 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1679 bytes)
I0223 01:15:07.145802 662830 provision.go:86] duration metric: configureAuth took 287.870729ms
I0223 01:15:07.145841 662830 ubuntu.go:193] setting minikube options for container-runtime
I0223 01:15:07.146006 662830 config.go:182] Loaded profile config "old-k8s-version-799707": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.16.0
I0223 01:15:07.146090 662830 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-799707
I0223 01:15:07.163391 662830 main.go:141] libmachine: Using SSH client type: native
I0223 01:15:07.163586 662830 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x82d280] 0x82ffe0 <nil> [] 0s} 127.0.0.1 33358 <nil> <nil>}
I0223 01:15:07.163619 662830 main.go:141] libmachine: About to run SSH command:
df --output=fstype / | tail -n 1
I0223 01:15:07.298676 662830 main.go:141] libmachine: SSH cmd err, output: <nil>: overlay
I0223 01:15:07.298702 662830 ubuntu.go:71] root file system type: overlay
I0223 01:15:07.298836 662830 provision.go:309] Updating docker unit: /lib/systemd/system/docker.service ...
I0223 01:15:07.298900 662830 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-799707
I0223 01:15:07.319323 662830 main.go:141] libmachine: Using SSH client type: native
I0223 01:15:07.319598 662830 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x82d280] 0x82ffe0 <nil> [] 0s} 127.0.0.1 33358 <nil> <nil>}
I0223 01:15:07.319710 662830 main.go:141] libmachine: About to run SSH command:
sudo mkdir -p /lib/systemd/system && printf %s "[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP \$MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service.new
I0223 01:15:07.465805 662830 main.go:141] libmachine: SSH cmd err, output: <nil>: [Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
I0223 01:15:07.465915 662830 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-799707
I0223 01:15:07.483319 662830 main.go:141] libmachine: Using SSH client type: native
I0223 01:15:07.483559 662830 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x82d280] 0x82ffe0 <nil> [] 0s} 127.0.0.1 33358 <nil> <nil>}
I0223 01:15:07.483586 662830 main.go:141] libmachine: About to run SSH command:
sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
I0223 01:15:08.223837 662830 main.go:141] libmachine: SSH cmd err, output: <nil>: --- /lib/systemd/system/docker.service 2024-02-06 21:12:51.000000000 +0000
+++ /lib/systemd/system/docker.service.new 2024-02-23 01:15:07.458899994 +0000
@@ -1,46 +1,49 @@
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
-After=network-online.target docker.socket firewalld.service containerd.service time-set.target
-Wants=network-online.target containerd.service
+BindsTo=containerd.service
+After=network-online.target firewalld.service containerd.service
+Wants=network-online.target
Requires=docker.socket
+StartLimitBurst=3
+StartLimitIntervalSec=60
[Service]
Type=notify
-# the default is not to use systemd for cgroups because the delegate issues still
-# exists and systemd currently does not support the cgroup feature set required
-# for containers run by docker
-ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
-ExecReload=/bin/kill -s HUP $MAINPID
-TimeoutStartSec=0
-RestartSec=2
-Restart=always
+Restart=on-failure
-# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
-# Both the old, and new location are accepted by systemd 229 and up, so using the old location
-# to make them work for either version of systemd.
-StartLimitBurst=3
-# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
-# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
-# this option work for either version of systemd.
-StartLimitInterval=60s
+
+# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
+# The base configuration already specifies an 'ExecStart=...' command. The first directive
+# here is to clear out that command inherited from the base configuration. Without this,
+# the command from the base configuration and the command specified here are treated as
+# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
+# will catch this invalid input and refuse to start the service with an error like:
+# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
+
+# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
+# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
+ExecStart=
+ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
+ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
+LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
-# Comment TasksMax if your systemd version does not support it.
-# Only systemd 226 and above support this option.
+# Uncomment TasksMax if your systemd version supports it.
+# Only systemd 226 and above support this version.
TasksMax=infinity
+TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
-OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
I0223 01:15:08.223873 662830 machine.go:91] provisioned docker machine in 1.705695788s
I0223 01:15:08.223883 662830 client.go:171] LocalClient.Create took 11.253775841s
I0223 01:15:08.223902 662830 start.go:167] duration metric: libmachine.API.Create for "old-k8s-version-799707" took 11.253862282s
I0223 01:15:08.223916 662830 start.go:300] post-start starting for "old-k8s-version-799707" (driver="docker")
I0223 01:15:08.223926 662830 start.go:329] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0223 01:15:08.223978 662830 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0223 01:15:08.224013 662830 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-799707
I0223 01:15:08.240944 662830 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33358 SSHKeyPath:/home/jenkins/minikube-integration/18233-317564/.minikube/machines/old-k8s-version-799707/id_rsa Username:docker}
I0223 01:15:08.334982 662830 ssh_runner.go:195] Run: cat /etc/os-release
I0223 01:15:08.338100 662830 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0223 01:15:08.338131 662830 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0223 01:15:08.338140 662830 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0223 01:15:08.338147 662830 info.go:137] Remote host: Ubuntu 22.04.3 LTS
I0223 01:15:08.338161 662830 filesync.go:126] Scanning /home/jenkins/minikube-integration/18233-317564/.minikube/addons for local assets ...
I0223 01:15:08.338209 662830 filesync.go:126] Scanning /home/jenkins/minikube-integration/18233-317564/.minikube/files for local assets ...
I0223 01:15:08.338284 662830 filesync.go:149] local asset: /home/jenkins/minikube-integration/18233-317564/.minikube/files/etc/ssl/certs/3243752.pem -> 3243752.pem in /etc/ssl/certs
I0223 01:15:08.338380 662830 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0223 01:15:08.346102 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/files/etc/ssl/certs/3243752.pem --> /etc/ssl/certs/3243752.pem (1708 bytes)
I0223 01:15:08.367148 662830 start.go:303] post-start completed in 143.218996ms
I0223 01:15:08.367490 662830 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-799707
I0223 01:15:08.385271 662830 profile.go:148] Saving config to /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/config.json ...
I0223 01:15:08.385589 662830 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0223 01:15:08.385643 662830 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-799707
I0223 01:15:08.402182 662830 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33358 SSHKeyPath:/home/jenkins/minikube-integration/18233-317564/.minikube/machines/old-k8s-version-799707/id_rsa Username:docker}
I0223 01:15:08.490731 662830 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0223 01:15:08.494805 662830 start.go:128] duration metric: createHost completed in 11.531090105s
I0223 01:15:08.494831 662830 start.go:83] releasing machines lock for "old-k8s-version-799707", held for 11.531304892s
I0223 01:15:08.494921 662830 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-799707
I0223 01:15:08.510988 662830 ssh_runner.go:195] Run: cat /version.json
I0223 01:15:08.511051 662830 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-799707
I0223 01:15:08.511052 662830 ssh_runner.go:195] Run: curl -sS -m 2 https://registry.k8s.io/
I0223 01:15:08.511129 662830 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-799707
I0223 01:15:08.529648 662830 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33358 SSHKeyPath:/home/jenkins/minikube-integration/18233-317564/.minikube/machines/old-k8s-version-799707/id_rsa Username:docker}
I0223 01:15:08.529928 662830 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33358 SSHKeyPath:/home/jenkins/minikube-integration/18233-317564/.minikube/machines/old-k8s-version-799707/id_rsa Username:docker}
I0223 01:15:08.707198 662830 ssh_runner.go:195] Run: systemctl --version
I0223 01:15:08.711459 662830 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
I0223 01:15:08.715389 662830 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *loopback.conf* -not -name *.mk_disabled -exec sh -c "grep -q loopback {} && ( grep -q name {} || sudo sed -i '/"type": "loopback"/i \ \ \ \ "name": "loopback",' {} ) && sudo sed -i 's|"cniVersion": ".*"|"cniVersion": "1.0.0"|g' {}" ;
I0223 01:15:08.737393 662830 cni.go:230] loopback cni configuration patched: "/etc/cni/net.d/*loopback.conf*" found
I0223 01:15:08.737508 662830 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *bridge* -not -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e '/"dst": ".*:.*"/d' -e 's|^(.*)"dst": (.*)[,*]$|\1"dst": \2|g' -e '/"subnet": ".*:.*"/d' -e 's|^(.*)"subnet": ".*"(.*)[,*]$|\1"subnet": "10.244.0.0/16"\2|g' {}" ;
I0223 01:15:08.752300 662830 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e 's|^(.*)"subnet": ".*"(.*)$|\1"subnet": "10.244.0.0/16"\2|g' -e 's|^(.*)"gateway": ".*"(.*)$|\1"gateway": "10.244.0.1"\2|g' {}" ;
I0223 01:15:08.767741 662830 cni.go:308] configured [/etc/cni/net.d/100-crio-bridge.conf, /etc/cni/net.d/87-podman-bridge.conflist] bridge cni config(s)
I0223 01:15:08.767777 662830 start.go:475] detecting cgroup driver to use...
I0223 01:15:08.767815 662830 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0223 01:15:08.767992 662830 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0223 01:15:08.783639 662830 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.1"|' /etc/containerd/config.toml"
I0223 01:15:08.793147 662830 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I0223 01:15:08.804013 662830 containerd.go:146] configuring containerd to use "cgroupfs" as cgroup driver...
I0223 01:15:08.804074 662830 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = false|g' /etc/containerd/config.toml"
I0223 01:15:08.813466 662830 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0223 01:15:08.822714 662830 ssh_runner.go:195] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I0223 01:15:08.831548 662830 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0223 01:15:08.840005 662830 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0223 01:15:08.847827 662830 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I0223 01:15:08.856158 662830 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0223 01:15:08.863600 662830 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0223 01:15:08.870838 662830 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0223 01:15:08.937270 662830 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0223 01:15:09.073929 662830 start.go:475] detecting cgroup driver to use...
I0223 01:15:09.073986 662830 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0223 01:15:09.074076 662830 ssh_runner.go:195] Run: sudo systemctl cat docker.service
I0223 01:15:09.112087 662830 cruntime.go:279] skipping containerd shutdown because we are bound to it
I0223 01:15:09.112150 662830 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0223 01:15:09.123788 662830 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///var/run/dockershim.sock
" | sudo tee /etc/crictl.yaml"
I0223 01:15:09.140597 662830 ssh_runner.go:195] Run: which cri-dockerd
I0223 01:15:09.143994 662830 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/cri-docker.service.d
I0223 01:15:09.152511 662830 ssh_runner.go:362] scp memory --> /etc/systemd/system/cri-docker.service.d/10-cni.conf (189 bytes)
I0223 01:15:09.182762 662830 ssh_runner.go:195] Run: sudo systemctl unmask docker.service
I0223 01:15:09.256843 662830 ssh_runner.go:195] Run: sudo systemctl enable docker.socket
I0223 01:15:09.370622 662830 docker.go:574] configuring docker to use "cgroupfs" as cgroup driver...
I0223 01:15:09.370817 662830 ssh_runner.go:362] scp memory --> /etc/docker/daemon.json (130 bytes)
I0223 01:15:09.389613 662830 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0223 01:15:09.466515 662830 ssh_runner.go:195] Run: sudo systemctl restart docker
I0223 01:15:09.853163 662830 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0223 01:15:09.882041 662830 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0223 01:15:09.910231 662830 out.go:204] * Preparing Kubernetes v1.16.0 on Docker 25.0.3 ...
I0223 01:15:09.910319 662830 cli_runner.go:164] Run: docker network inspect old-k8s-version-799707 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0223 01:15:09.932561 662830 ssh_runner.go:195] Run: grep 192.168.94.1 host.minikube.internal$ /etc/hosts
I0223 01:15:09.936890 662830 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.94.1 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0223 01:15:09.948140 662830 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0223 01:15:09.948192 662830 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0223 01:15:09.970748 662830 docker.go:685] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0223 01:15:09.970773 662830 docker.go:691] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0223 01:15:09.970826 662830 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0223 01:15:09.981424 662830 ssh_runner.go:195] Run: which lz4
I0223 01:15:09.984840 662830 ssh_runner.go:195] Run: stat -c "%s %y" /preloaded.tar.lz4
I0223 01:15:09.988470 662830 ssh_runner.go:352] existence check for /preloaded.tar.lz4: stat -c "%s %y" /preloaded.tar.lz4: Process exited with status 1
stdout:
stderr:
stat: cannot statx '/preloaded.tar.lz4': No such file or directory
I0223 01:15:09.988499 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 --> /preloaded.tar.lz4 (369789069 bytes)
I0223 01:15:10.924525 662830 docker.go:649] Took 0.939727 seconds to copy over tarball
I0223 01:15:10.924605 662830 ssh_runner.go:195] Run: sudo tar --xattrs --xattrs-include security.capability -I lz4 -C /var -xf /preloaded.tar.lz4
I0223 01:15:13.443394 662830 ssh_runner.go:235] Completed: sudo tar --xattrs --xattrs-include security.capability -I lz4 -C /var -xf /preloaded.tar.lz4: (2.518754917s)
I0223 01:15:13.443428 662830 ssh_runner.go:146] rm: /preloaded.tar.lz4
I0223 01:15:13.502931 662830 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0223 01:15:13.511214 662830 ssh_runner.go:362] scp memory --> /var/lib/docker/image/overlay2/repositories.json (2499 bytes)
I0223 01:15:13.527242 662830 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0223 01:15:13.601728 662830 ssh_runner.go:195] Run: sudo systemctl restart docker
I0223 01:15:14.339608 662830 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0223 01:15:14.361618 662830 docker.go:685] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0223 01:15:14.361641 662830 docker.go:691] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0223 01:15:14.361653 662830 cache_images.go:88] LoadImages start: [registry.k8s.io/kube-apiserver:v1.16.0 registry.k8s.io/kube-controller-manager:v1.16.0 registry.k8s.io/kube-scheduler:v1.16.0 registry.k8s.io/kube-proxy:v1.16.0 registry.k8s.io/pause:3.1 registry.k8s.io/etcd:3.3.15-0 registry.k8s.io/coredns:1.6.2 gcr.io/k8s-minikube/storage-provisioner:v5]
I0223 01:15:14.363181 662830 image.go:134] retrieving image: registry.k8s.io/kube-proxy:v1.16.0
I0223 01:15:14.363468 662830 image.go:134] retrieving image: registry.k8s.io/kube-apiserver:v1.16.0
I0223 01:15:14.363506 662830 image.go:134] retrieving image: registry.k8s.io/etcd:3.3.15-0
I0223 01:15:14.363555 662830 image.go:134] retrieving image: registry.k8s.io/pause:3.1
I0223 01:15:14.363598 662830 image.go:134] retrieving image: registry.k8s.io/kube-scheduler:v1.16.0
I0223 01:15:14.363657 662830 image.go:134] retrieving image: registry.k8s.io/kube-controller-manager:v1.16.0
I0223 01:15:14.363689 662830 image.go:134] retrieving image: gcr.io/k8s-minikube/storage-provisioner:v5
I0223 01:15:14.363728 662830 image.go:134] retrieving image: registry.k8s.io/coredns:1.6.2
I0223 01:15:14.364142 662830 image.go:177] daemon lookup for registry.k8s.io/kube-proxy:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-proxy:v1.16.0
I0223 01:15:14.364381 662830 image.go:177] daemon lookup for registry.k8s.io/kube-scheduler:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-scheduler:v1.16.0
I0223 01:15:14.364443 662830 image.go:177] daemon lookup for registry.k8s.io/kube-apiserver:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-apiserver:v1.16.0
I0223 01:15:14.364517 662830 image.go:177] daemon lookup for registry.k8s.io/pause:3.1: Error response from daemon: No such image: registry.k8s.io/pause:3.1
I0223 01:15:14.364673 662830 image.go:177] daemon lookup for gcr.io/k8s-minikube/storage-provisioner:v5: Error response from daemon: No such image: gcr.io/k8s-minikube/storage-provisioner:v5
I0223 01:15:14.364686 662830 image.go:177] daemon lookup for registry.k8s.io/etcd:3.3.15-0: Error response from daemon: No such image: registry.k8s.io/etcd:3.3.15-0
I0223 01:15:14.364797 662830 image.go:177] daemon lookup for registry.k8s.io/kube-controller-manager:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-controller-manager:v1.16.0
I0223 01:15:14.364920 662830 image.go:177] daemon lookup for registry.k8s.io/coredns:1.6.2: Error response from daemon: No such image: registry.k8s.io/coredns:1.6.2
I0223 01:15:14.561683 662830 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-controller-manager:v1.16.0
I0223 01:15:14.561683 662830 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/etcd:3.3.15-0
I0223 01:15:14.564037 662830 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/pause:3.1
I0223 01:15:14.571035 662830 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/coredns:1.6.2
I0223 01:15:14.573935 662830 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} gcr.io/k8s-minikube/storage-provisioner:v5
I0223 01:15:14.598601 662830 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-scheduler:v1.16.0
I0223 01:15:14.598970 662830 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-proxy:v1.16.0
I0223 01:15:14.599149 662830 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-apiserver:v1.16.0
I0223 01:15:14.604824 662830 cache_images.go:116] "registry.k8s.io/etcd:3.3.15-0" needs transfer: "registry.k8s.io/etcd:3.3.15-0" does not exist at hash "b2756210eeabf84f3221da9959e9483f3919dc2aaab4cd45e7cd072fcbde27ed" in container runtime
I0223 01:15:14.604884 662830 docker.go:337] Removing image: registry.k8s.io/etcd:3.3.15-0
I0223 01:15:14.604903 662830 cache_images.go:116] "registry.k8s.io/kube-controller-manager:v1.16.0" needs transfer: "registry.k8s.io/kube-controller-manager:v1.16.0" does not exist at hash "06a629a7e51cdcc81a5ed6a3e6650348312f20c954ac52ee489a023628ec9c7d" in container runtime
I0223 01:15:14.604921 662830 ssh_runner.go:195] Run: docker rmi registry.k8s.io/etcd:3.3.15-0
I0223 01:15:14.604976 662830 docker.go:337] Removing image: registry.k8s.io/kube-controller-manager:v1.16.0
I0223 01:15:14.605017 662830 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-controller-manager:v1.16.0
I0223 01:15:14.679559 662830 cache_images.go:116] "registry.k8s.io/coredns:1.6.2" needs transfer: "registry.k8s.io/coredns:1.6.2" does not exist at hash "bf261d157914477ee1a5969d28ec687f3fbfc9fb5a664b22df78e57023b0e03b" in container runtime
I0223 01:15:14.679621 662830 docker.go:337] Removing image: registry.k8s.io/coredns:1.6.2
I0223 01:15:14.679665 662830 ssh_runner.go:195] Run: docker rmi registry.k8s.io/coredns:1.6.2
I0223 01:15:14.679772 662830 cache_images.go:116] "registry.k8s.io/kube-apiserver:v1.16.0" needs transfer: "registry.k8s.io/kube-apiserver:v1.16.0" does not exist at hash "b305571ca60a5a7818bda47da122683d75e8a1907475681ee8b1efbd06bff12e" in container runtime
I0223 01:15:14.679801 662830 docker.go:337] Removing image: registry.k8s.io/kube-apiserver:v1.16.0
I0223 01:15:14.679829 662830 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-apiserver:v1.16.0
I0223 01:15:14.679882 662830 cache_images.go:116] "registry.k8s.io/kube-scheduler:v1.16.0" needs transfer: "registry.k8s.io/kube-scheduler:v1.16.0" does not exist at hash "301ddc62b80b16315d3c2653cf3888370394277afb3187614cfa20edc352ca0a" in container runtime
I0223 01:15:14.679911 662830 docker.go:337] Removing image: registry.k8s.io/kube-scheduler:v1.16.0
I0223 01:15:14.679944 662830 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-scheduler:v1.16.0
I0223 01:15:14.680035 662830 cache_images.go:116] "registry.k8s.io/kube-proxy:v1.16.0" needs transfer: "registry.k8s.io/kube-proxy:v1.16.0" does not exist at hash "c21b0c7400f988db4777858edd13b6d3930d62d7ccf026d2415485a52037f384" in container runtime
I0223 01:15:14.680078 662830 docker.go:337] Removing image: registry.k8s.io/kube-proxy:v1.16.0
I0223 01:15:14.680120 662830 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-proxy:v1.16.0
I0223 01:15:14.680199 662830 cache_images.go:116] "registry.k8s.io/pause:3.1" needs transfer: "registry.k8s.io/pause:3.1" does not exist at hash "da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e" in container runtime
I0223 01:15:14.680225 662830 docker.go:337] Removing image: registry.k8s.io/pause:3.1
I0223 01:15:14.680251 662830 ssh_runner.go:195] Run: docker rmi registry.k8s.io/pause:3.1
I0223 01:15:14.680484 662830 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/18233-317564/.minikube/cache/images/amd64/registry.k8s.io/etcd_3.3.15-0
I0223 01:15:14.680564 662830 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/18233-317564/.minikube/cache/images/amd64/registry.k8s.io/kube-controller-manager_v1.16.0
I0223 01:15:14.707372 662830 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/18233-317564/.minikube/cache/images/amd64/registry.k8s.io/coredns_1.6.2
I0223 01:15:14.707417 662830 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/18233-317564/.minikube/cache/images/amd64/registry.k8s.io/kube-proxy_v1.16.0
I0223 01:15:14.716153 662830 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/18233-317564/.minikube/cache/images/amd64/registry.k8s.io/pause_3.1
I0223 01:15:14.716228 662830 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/18233-317564/.minikube/cache/images/amd64/registry.k8s.io/kube-scheduler_v1.16.0
I0223 01:15:14.716488 662830 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/18233-317564/.minikube/cache/images/amd64/registry.k8s.io/kube-apiserver_v1.16.0
I0223 01:15:14.716545 662830 cache_images.go:92] LoadImages completed in 354.87672ms
W0223 01:15:14.716632 662830 out.go:239] X Unable to load cached images: loading cached images: stat /home/jenkins/minikube-integration/18233-317564/.minikube/cache/images/amd64/registry.k8s.io/etcd_3.3.15-0: no such file or directory
X Unable to load cached images: loading cached images: stat /home/jenkins/minikube-integration/18233-317564/.minikube/cache/images/amd64/registry.k8s.io/etcd_3.3.15-0: no such file or directory
I0223 01:15:14.716705 662830 ssh_runner.go:195] Run: docker info --format {{.CgroupDriver}}
I0223 01:15:14.807172 662830 cni.go:84] Creating CNI manager for ""
I0223 01:15:14.807193 662830 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0223 01:15:14.807211 662830 kubeadm.go:87] Using pod CIDR: 10.244.0.0/16
I0223 01:15:14.807227 662830 kubeadm.go:176] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.94.2 APIServerPort:8443 KubernetesVersion:v1.16.0 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:old-k8s-version-799707 NodeName:old-k8s-version-799707 DNSDomain:cluster.local CRISocket:/var/run/dockershim.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.94.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.94.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticP
odPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[hairpinMode:hairpin-veth runtimeRequestTimeout:15m] PrependCriSocketUnix:false}
I0223 01:15:14.807374 662830 kubeadm.go:181] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.94.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: "old-k8s-version-799707"
kubeletExtraArgs:
node-ip: 192.168.94.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.94.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: old-k8s-version-799707
controlPlaneEndpoint: control-plane.minikube.internal:8443
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
listen-metrics-urls: http://127.0.0.1:2381,http://192.168.94.2:2381
kubernetesVersion: v1.16.0
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0223 01:15:14.807465 662830 kubeadm.go:976] kubelet [Unit]
Wants=docker.socket
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.16.0/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=old-k8s-version-799707 --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.94.2
[Install]
config:
{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-799707 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:}
I0223 01:15:14.807511 662830 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.16.0
I0223 01:15:14.816205 662830 binaries.go:44] Found k8s binaries, skipping transfer
I0223 01:15:14.816263 662830 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0223 01:15:14.860133 662830 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (348 bytes)
I0223 01:15:14.897670 662830 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0223 01:15:14.917909 662830 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2174 bytes)
I0223 01:15:14.937050 662830 ssh_runner.go:195] Run: grep 192.168.94.2 control-plane.minikube.internal$ /etc/hosts
I0223 01:15:14.940699 662830 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.94.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0223 01:15:14.952831 662830 certs.go:56] Setting up /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707 for IP: 192.168.94.2
I0223 01:15:14.952881 662830 certs.go:190] acquiring lock for shared ca certs: {Name:mk61b7180586719fd962a2bfdb44a8ad933bd3aa Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0223 01:15:14.953180 662830 certs.go:199] skipping minikubeCA CA generation: /home/jenkins/minikube-integration/18233-317564/.minikube/ca.key
I0223 01:15:14.953235 662830 certs.go:199] skipping proxyClientCA CA generation: /home/jenkins/minikube-integration/18233-317564/.minikube/proxy-client-ca.key
I0223 01:15:14.953299 662830 certs.go:319] generating minikube-user signed cert: /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/client.key
I0223 01:15:14.953356 662830 crypto.go:68] Generating cert /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/client.crt with IP's: []
I0223 01:15:15.345958 662830 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/client.crt ...
I0223 01:15:15.345993 662830 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/client.crt: {Name:mke193cfa64c426bc0ba5afa11c4d2c28013bbad Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0223 01:15:15.346184 662830 crypto.go:164] Writing key to /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/client.key ...
I0223 01:15:15.346201 662830 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/client.key: {Name:mk3df5dd67d7b375296104968c6e3b5a4be9be7a Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0223 01:15:15.346303 662830 certs.go:319] generating minikube signed cert: /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/apiserver.key.ad8e880a
I0223 01:15:15.346317 662830 crypto.go:68] Generating cert /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/apiserver.crt.ad8e880a with IP's: [192.168.94.2 10.96.0.1 127.0.0.1 10.0.0.1]
I0223 01:15:15.751447 662830 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/apiserver.crt.ad8e880a ...
I0223 01:15:15.751488 662830 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/apiserver.crt.ad8e880a: {Name:mk8e4b8bf5412a1da63b54b677d9b8c1d623a4a2 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0223 01:15:15.751720 662830 crypto.go:164] Writing key to /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/apiserver.key.ad8e880a ...
I0223 01:15:15.751737 662830 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/apiserver.key.ad8e880a: {Name:mk045124f08a36c842462ba538d49dbfc2b8c2f0 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0223 01:15:15.751859 662830 certs.go:337] copying /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/apiserver.crt.ad8e880a -> /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/apiserver.crt
I0223 01:15:15.751951 662830 certs.go:341] copying /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/apiserver.key.ad8e880a -> /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/apiserver.key
I0223 01:15:15.752017 662830 certs.go:319] generating aggregator signed cert: /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/proxy-client.key
I0223 01:15:15.752042 662830 crypto.go:68] Generating cert /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/proxy-client.crt with IP's: []
I0223 01:15:15.996126 662830 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/proxy-client.crt ...
I0223 01:15:15.996166 662830 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/proxy-client.crt: {Name:mk871f69f10bc41b5f295bc85ac3b1cae9c1a71c Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0223 01:15:15.996361 662830 crypto.go:164] Writing key to /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/proxy-client.key ...
I0223 01:15:15.996380 662830 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/proxy-client.key: {Name:mk0f74328cdf485742cad6f0f077781f34161152 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0223 01:15:15.996642 662830 certs.go:437] found cert: /home/jenkins/minikube-integration/18233-317564/.minikube/certs/home/jenkins/minikube-integration/18233-317564/.minikube/certs/324375.pem (1338 bytes)
W0223 01:15:15.996689 662830 certs.go:433] ignoring /home/jenkins/minikube-integration/18233-317564/.minikube/certs/home/jenkins/minikube-integration/18233-317564/.minikube/certs/324375_empty.pem, impossibly tiny 0 bytes
I0223 01:15:15.996707 662830 certs.go:437] found cert: /home/jenkins/minikube-integration/18233-317564/.minikube/certs/home/jenkins/minikube-integration/18233-317564/.minikube/certs/ca-key.pem (1679 bytes)
I0223 01:15:15.996745 662830 certs.go:437] found cert: /home/jenkins/minikube-integration/18233-317564/.minikube/certs/home/jenkins/minikube-integration/18233-317564/.minikube/certs/ca.pem (1078 bytes)
I0223 01:15:15.996777 662830 certs.go:437] found cert: /home/jenkins/minikube-integration/18233-317564/.minikube/certs/home/jenkins/minikube-integration/18233-317564/.minikube/certs/cert.pem (1123 bytes)
I0223 01:15:15.996808 662830 certs.go:437] found cert: /home/jenkins/minikube-integration/18233-317564/.minikube/certs/home/jenkins/minikube-integration/18233-317564/.minikube/certs/key.pem (1675 bytes)
I0223 01:15:15.996878 662830 certs.go:437] found cert: /home/jenkins/minikube-integration/18233-317564/.minikube/files/etc/ssl/certs/home/jenkins/minikube-integration/18233-317564/.minikube/files/etc/ssl/certs/3243752.pem (1708 bytes)
I0223 01:15:15.997549 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1399 bytes)
I0223 01:15:16.020672 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1679 bytes)
I0223 01:15:16.043235 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0223 01:15:16.064664 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/profiles/old-k8s-version-799707/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1679 bytes)
I0223 01:15:16.092606 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0223 01:15:16.114466 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1675 bytes)
I0223 01:15:16.136498 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0223 01:15:16.158821 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1675 bytes)
I0223 01:15:16.184715 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/certs/324375.pem --> /usr/share/ca-certificates/324375.pem (1338 bytes)
I0223 01:15:16.206463 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/files/etc/ssl/certs/3243752.pem --> /usr/share/ca-certificates/3243752.pem (1708 bytes)
I0223 01:15:16.228992 662830 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18233-317564/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0223 01:15:16.254783 662830 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0223 01:15:16.270850 662830 ssh_runner.go:195] Run: openssl version
I0223 01:15:16.275820 662830 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/324375.pem && ln -fs /usr/share/ca-certificates/324375.pem /etc/ssl/certs/324375.pem"
I0223 01:15:16.284181 662830 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/324375.pem
I0223 01:15:16.287717 662830 certs.go:480] hashing: -rw-r--r-- 1 root root 1338 Feb 23 00:36 /usr/share/ca-certificates/324375.pem
I0223 01:15:16.287779 662830 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/324375.pem
I0223 01:15:16.294282 662830 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/324375.pem /etc/ssl/certs/51391683.0"
I0223 01:15:16.303486 662830 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/3243752.pem && ln -fs /usr/share/ca-certificates/3243752.pem /etc/ssl/certs/3243752.pem"
I0223 01:15:16.313046 662830 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/3243752.pem
I0223 01:15:16.316579 662830 certs.go:480] hashing: -rw-r--r-- 1 root root 1708 Feb 23 00:36 /usr/share/ca-certificates/3243752.pem
I0223 01:15:16.316639 662830 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/3243752.pem
I0223 01:15:16.322962 662830 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/3243752.pem /etc/ssl/certs/3ec20f2e.0"
I0223 01:15:16.332026 662830 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0223 01:15:16.340686 662830 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0223 01:15:16.343904 662830 certs.go:480] hashing: -rw-r--r-- 1 root root 1111 Feb 23 00:32 /usr/share/ca-certificates/minikubeCA.pem
I0223 01:15:16.343967 662830 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0223 01:15:16.351156 662830 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0223 01:15:16.363435 662830 ssh_runner.go:195] Run: ls /var/lib/minikube/certs/etcd
I0223 01:15:16.366876 662830 certs.go:353] certs directory doesn't exist, likely first start: ls /var/lib/minikube/certs/etcd: Process exited with status 2
stdout:
stderr:
ls: cannot access '/var/lib/minikube/certs/etcd': No such file or directory
I0223 01:15:16.366934 662830 kubeadm.go:404] StartCluster: {Name:old-k8s-version-799707 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-799707 Namespace:default APIServerName:minikubeCA APISer
verNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.94.2 Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:fals
e DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0223 01:15:16.367049 662830 ssh_runner.go:195] Run: docker ps --filter status=paused --filter=name=k8s_.*_(kube-system)_ --format={{.ID}}
I0223 01:15:16.388038 662830 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0223 01:15:16.397123 662830 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0223 01:15:16.405858 662830 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0223 01:15:16.405920 662830 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0223 01:15:16.413713 662830 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0223 01:15:16.413759 662830 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0223 01:15:16.479330 662830 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0223 01:15:16.479394 662830 kubeadm.go:322] [preflight] Running pre-flight checks
I0223 01:15:16.676898 662830 kubeadm.go:322] [preflight] The system verification failed. Printing the output from the verification:
I0223 01:15:16.676971 662830 kubeadm.go:322] [0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
I0223 01:15:16.677011 662830 kubeadm.go:322] [0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
I0223 01:15:16.677041 662830 kubeadm.go:322] [0;37mOS[0m: [0;32mLinux[0m
I0223 01:15:16.677079 662830 kubeadm.go:322] [0;37mCGROUPS_CPU[0m: [0;32menabled[0m
I0223 01:15:16.677120 662830 kubeadm.go:322] [0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
I0223 01:15:16.677159 662830 kubeadm.go:322] [0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
I0223 01:15:16.677200 662830 kubeadm.go:322] [0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
I0223 01:15:16.677247 662830 kubeadm.go:322] [0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
I0223 01:15:16.677298 662830 kubeadm.go:322] [0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
I0223 01:15:16.764811 662830 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0223 01:15:16.764957 662830 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0223 01:15:16.765086 662830 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0223 01:15:16.954902 662830 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0223 01:15:16.956631 662830 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0223 01:15:16.968887 662830 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0223 01:15:17.049621 662830 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0223 01:15:17.053037 662830 out.go:204] - Generating certificates and keys ...
I0223 01:15:17.053157 662830 kubeadm.go:322] [certs] Using existing ca certificate authority
I0223 01:15:17.053250 662830 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0223 01:15:17.525720 662830 kubeadm.go:322] [certs] Generating "apiserver-kubelet-client" certificate and key
I0223 01:15:17.801446 662830 kubeadm.go:322] [certs] Generating "front-proxy-ca" certificate and key
I0223 01:15:17.904332 662830 kubeadm.go:322] [certs] Generating "front-proxy-client" certificate and key
I0223 01:15:18.347321 662830 kubeadm.go:322] [certs] Generating "etcd/ca" certificate and key
I0223 01:15:18.437131 662830 kubeadm.go:322] [certs] Generating "etcd/server" certificate and key
I0223 01:15:18.438017 662830 kubeadm.go:322] [certs] etcd/server serving cert is signed for DNS names [old-k8s-version-799707 localhost] and IPs [192.168.94.2 127.0.0.1 ::1]
I0223 01:15:18.594938 662830 kubeadm.go:322] [certs] Generating "etcd/peer" certificate and key
I0223 01:15:18.595298 662830 kubeadm.go:322] [certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-799707 localhost] and IPs [192.168.94.2 127.0.0.1 ::1]
I0223 01:15:18.787280 662830 kubeadm.go:322] [certs] Generating "etcd/healthcheck-client" certificate and key
I0223 01:15:19.094470 662830 kubeadm.go:322] [certs] Generating "apiserver-etcd-client" certificate and key
I0223 01:15:19.539331 662830 kubeadm.go:322] [certs] Generating "sa" key and public key
I0223 01:15:19.539526 662830 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0223 01:15:19.661048 662830 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0223 01:15:20.082140 662830 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0223 01:15:20.424562 662830 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0223 01:15:20.681911 662830 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0223 01:15:20.683465 662830 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0223 01:15:20.685909 662830 out.go:204] - Booting up control plane ...
I0223 01:15:20.686106 662830 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0223 01:15:20.699215 662830 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0223 01:15:20.700996 662830 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0223 01:15:20.702205 662830 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0223 01:15:20.705549 662830 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0223 01:16:00.705707 662830 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0223 01:19:20.707360 662830 kubeadm.go:322]
I0223 01:19:20.707537 662830 kubeadm.go:322] Unfortunately, an error has occurred:
I0223 01:19:20.707709 662830 kubeadm.go:322] timed out waiting for the condition
I0223 01:19:20.707759 662830 kubeadm.go:322]
I0223 01:19:20.707856 662830 kubeadm.go:322] This error is likely caused by:
I0223 01:19:20.707932 662830 kubeadm.go:322] - The kubelet is not running
I0223 01:19:20.708189 662830 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0223 01:19:20.708219 662830 kubeadm.go:322]
I0223 01:19:20.708403 662830 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0223 01:19:20.708476 662830 kubeadm.go:322] - 'systemctl status kubelet'
I0223 01:19:20.708539 662830 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0223 01:19:20.708548 662830 kubeadm.go:322]
I0223 01:19:20.708676 662830 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0223 01:19:20.708814 662830 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0223 01:19:20.708926 662830 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0223 01:19:20.708997 662830 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0223 01:19:20.709105 662830 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0223 01:19:20.709167 662830 kubeadm.go:322] - 'docker logs CONTAINERID'
I0223 01:19:20.711478 662830 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0223 01:19:20.711608 662830 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
I0223 01:19:20.711849 662830 kubeadm.go:322] [WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
I0223 01:19:20.711971 662830 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0223 01:19:20.712089 662830 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0223 01:19:20.712225 662830 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
W0223 01:19:20.712428 662830 out.go:239] ! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-799707 localhost] and IPs [192.168.94.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-799707 localhost] and IPs [192.168.94.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-799707 localhost] and IPs [192.168.94.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-799707 localhost] and IPs [192.168.94.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
I0223 01:19:20.712518 662830 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm reset --cri-socket /var/run/dockershim.sock --force"
I0223 01:19:21.476427 662830 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service kubelet
I0223 01:19:21.487973 662830 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0223 01:19:21.488031 662830 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0223 01:19:21.496281 662830 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0223 01:19:21.496330 662830 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0223 01:19:21.542866 662830 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0223 01:19:21.542965 662830 kubeadm.go:322] [preflight] Running pre-flight checks
I0223 01:19:21.718099 662830 kubeadm.go:322] [preflight] The system verification failed. Printing the output from the verification:
I0223 01:19:21.718190 662830 kubeadm.go:322] [0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
I0223 01:19:21.718250 662830 kubeadm.go:322] [0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
I0223 01:19:21.718298 662830 kubeadm.go:322] [0;37mOS[0m: [0;32mLinux[0m
I0223 01:19:21.718351 662830 kubeadm.go:322] [0;37mCGROUPS_CPU[0m: [0;32menabled[0m
I0223 01:19:21.718412 662830 kubeadm.go:322] [0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
I0223 01:19:21.718480 662830 kubeadm.go:322] [0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
I0223 01:19:21.718535 662830 kubeadm.go:322] [0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
I0223 01:19:21.718599 662830 kubeadm.go:322] [0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
I0223 01:19:21.718661 662830 kubeadm.go:322] [0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
I0223 01:19:21.788286 662830 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0223 01:19:21.788427 662830 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0223 01:19:21.788545 662830 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0223 01:19:21.964469 662830 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0223 01:19:21.965592 662830 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0223 01:19:21.972954 662830 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0223 01:19:22.055594 662830 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0223 01:19:22.057674 662830 out.go:204] - Generating certificates and keys ...
I0223 01:19:22.057783 662830 kubeadm.go:322] [certs] Using existing ca certificate authority
I0223 01:19:22.057869 662830 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0223 01:19:22.058016 662830 kubeadm.go:322] [certs] Using existing apiserver-kubelet-client certificate and key on disk
I0223 01:19:22.058141 662830 kubeadm.go:322] [certs] Using existing front-proxy-ca certificate authority
I0223 01:19:22.058237 662830 kubeadm.go:322] [certs] Using existing front-proxy-client certificate and key on disk
I0223 01:19:22.058314 662830 kubeadm.go:322] [certs] Using existing etcd/ca certificate authority
I0223 01:19:22.058419 662830 kubeadm.go:322] [certs] Using existing etcd/server certificate and key on disk
I0223 01:19:22.058501 662830 kubeadm.go:322] [certs] Using existing etcd/peer certificate and key on disk
I0223 01:19:22.058573 662830 kubeadm.go:322] [certs] Using existing etcd/healthcheck-client certificate and key on disk
I0223 01:19:22.058898 662830 kubeadm.go:322] [certs] Using existing apiserver-etcd-client certificate and key on disk
I0223 01:19:22.058965 662830 kubeadm.go:322] [certs] Using the existing "sa" key
I0223 01:19:22.059040 662830 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0223 01:19:22.240393 662830 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0223 01:19:22.397081 662830 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0223 01:19:22.721666 662830 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0223 01:19:22.857798 662830 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0223 01:19:22.858680 662830 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0223 01:19:22.861014 662830 out.go:204] - Booting up control plane ...
I0223 01:19:22.861132 662830 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0223 01:19:22.864069 662830 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0223 01:19:22.865223 662830 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0223 01:19:22.865964 662830 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0223 01:19:22.870144 662830 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0223 01:20:02.870361 662830 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0223 01:23:22.871767 662830 kubeadm.go:322]
I0223 01:23:22.872047 662830 kubeadm.go:322] Unfortunately, an error has occurred:
I0223 01:23:22.872151 662830 kubeadm.go:322] timed out waiting for the condition
I0223 01:23:22.872169 662830 kubeadm.go:322]
I0223 01:23:22.872236 662830 kubeadm.go:322] This error is likely caused by:
I0223 01:23:22.872314 662830 kubeadm.go:322] - The kubelet is not running
I0223 01:23:22.872550 662830 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0223 01:23:22.872565 662830 kubeadm.go:322]
I0223 01:23:22.872786 662830 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0223 01:23:22.872856 662830 kubeadm.go:322] - 'systemctl status kubelet'
I0223 01:23:22.872926 662830 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0223 01:23:22.872937 662830 kubeadm.go:322]
I0223 01:23:22.873164 662830 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0223 01:23:22.873369 662830 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0223 01:23:22.873555 662830 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0223 01:23:22.873661 662830 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0223 01:23:22.873827 662830 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0223 01:23:22.873898 662830 kubeadm.go:322] - 'docker logs CONTAINERID'
I0223 01:23:22.875441 662830 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0223 01:23:22.875642 662830 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
I0223 01:23:22.875896 662830 kubeadm.go:322] [WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
I0223 01:23:22.876054 662830 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0223 01:23:22.876160 662830 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0223 01:23:22.876234 662830 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
I0223 01:23:22.876324 662830 kubeadm.go:406] StartCluster complete in 8m6.509401758s
I0223 01:23:22.876419 662830 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-apiserver --format={{.ID}}
I0223 01:23:22.895590 662830 logs.go:276] 0 containers: []
W0223 01:23:22.895614 662830 logs.go:278] No container was found matching "kube-apiserver"
I0223 01:23:22.895660 662830 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_etcd --format={{.ID}}
I0223 01:23:22.913998 662830 logs.go:276] 0 containers: []
W0223 01:23:22.914022 662830 logs.go:278] No container was found matching "etcd"
I0223 01:23:22.914096 662830 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_coredns --format={{.ID}}
I0223 01:23:22.931699 662830 logs.go:276] 0 containers: []
W0223 01:23:22.931729 662830 logs.go:278] No container was found matching "coredns"
I0223 01:23:22.931772 662830 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-scheduler --format={{.ID}}
I0223 01:23:22.948413 662830 logs.go:276] 0 containers: []
W0223 01:23:22.948445 662830 logs.go:278] No container was found matching "kube-scheduler"
I0223 01:23:22.948502 662830 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-proxy --format={{.ID}}
I0223 01:23:22.967459 662830 logs.go:276] 0 containers: []
W0223 01:23:22.967488 662830 logs.go:278] No container was found matching "kube-proxy"
I0223 01:23:22.967543 662830 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-controller-manager --format={{.ID}}
I0223 01:23:22.984821 662830 logs.go:276] 0 containers: []
W0223 01:23:22.984847 662830 logs.go:278] No container was found matching "kube-controller-manager"
I0223 01:23:22.984909 662830 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kindnet --format={{.ID}}
I0223 01:23:23.001713 662830 logs.go:276] 0 containers: []
W0223 01:23:23.001741 662830 logs.go:278] No container was found matching "kindnet"
I0223 01:23:23.001755 662830 logs.go:123] Gathering logs for container status ...
I0223 01:23:23.001771 662830 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
I0223 01:23:23.038578 662830 logs.go:123] Gathering logs for kubelet ...
I0223 01:23:23.038605 662830 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0223 01:23:23.062489 662830 logs.go:138] Found kubelet problem: Feb 23 01:23:02 old-k8s-version-799707 kubelet[5695]: E0223 01:23:02.555197 5695 pod_workers.go:191] Error syncing pod b3d303074fe0ca1d42a8bd9ed248df09 ("kube-scheduler-old-k8s-version-799707_kube-system(b3d303074fe0ca1d42a8bd9ed248df09)"), skipping: failed to "StartContainer" for "kube-scheduler" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-scheduler:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-scheduler:v1.16.0\" is not set"
W0223 01:23:23.068346 662830 logs.go:138] Found kubelet problem: Feb 23 01:23:05 old-k8s-version-799707 kubelet[5695]: E0223 01:23:05.555141 5695 pod_workers.go:191] Error syncing pod 9fc427d2e6746d2b3f18846f6f0fcafb ("kube-controller-manager-old-k8s-version-799707_kube-system(9fc427d2e6746d2b3f18846f6f0fcafb)"), skipping: failed to "StartContainer" for "kube-controller-manager" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-controller-manager:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-controller-manager:v1.16.0\" is not set"
W0223 01:23:23.070676 662830 logs.go:138] Found kubelet problem: Feb 23 01:23:06 old-k8s-version-799707 kubelet[5695]: E0223 01:23:06.555495 5695 pod_workers.go:191] Error syncing pod ada9a48d69d0428e9c15f5ad9c81ef07 ("etcd-old-k8s-version-799707_kube-system(ada9a48d69d0428e9c15f5ad9c81ef07)"), skipping: failed to "StartContainer" for "etcd" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/etcd:3.3.15-0\": Id or size of image \"k8s.gcr.io/etcd:3.3.15-0\" is not set"
W0223 01:23:23.081716 662830 logs.go:138] Found kubelet problem: Feb 23 01:23:12 old-k8s-version-799707 kubelet[5695]: E0223 01:23:12.560904 5695 pod_workers.go:191] Error syncing pod d7c490b6a42435ed7106e1a0ff029359 ("kube-apiserver-old-k8s-version-799707_kube-system(d7c490b6a42435ed7106e1a0ff029359)"), skipping: failed to "StartContainer" for "kube-apiserver" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-apiserver:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-apiserver:v1.16.0\" is not set"
W0223 01:23:23.090316 662830 logs.go:138] Found kubelet problem: Feb 23 01:23:17 old-k8s-version-799707 kubelet[5695]: E0223 01:23:17.558269 5695 pod_workers.go:191] Error syncing pod b3d303074fe0ca1d42a8bd9ed248df09 ("kube-scheduler-old-k8s-version-799707_kube-system(b3d303074fe0ca1d42a8bd9ed248df09)"), skipping: failed to "StartContainer" for "kube-scheduler" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-scheduler:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-scheduler:v1.16.0\" is not set"
W0223 01:23:23.090697 662830 logs.go:138] Found kubelet problem: Feb 23 01:23:17 old-k8s-version-799707 kubelet[5695]: E0223 01:23:17.559376 5695 pod_workers.go:191] Error syncing pod ada9a48d69d0428e9c15f5ad9c81ef07 ("etcd-old-k8s-version-799707_kube-system(ada9a48d69d0428e9c15f5ad9c81ef07)"), skipping: failed to "StartContainer" for "etcd" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/etcd:3.3.15-0\": Id or size of image \"k8s.gcr.io/etcd:3.3.15-0\" is not set"
W0223 01:23:23.096279 662830 logs.go:138] Found kubelet problem: Feb 23 01:23:20 old-k8s-version-799707 kubelet[5695]: E0223 01:23:20.555521 5695 pod_workers.go:191] Error syncing pod 9fc427d2e6746d2b3f18846f6f0fcafb ("kube-controller-manager-old-k8s-version-799707_kube-system(9fc427d2e6746d2b3f18846f6f0fcafb)"), skipping: failed to "StartContainer" for "kube-controller-manager" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-controller-manager:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-controller-manager:v1.16.0\" is not set"
I0223 01:23:23.100032 662830 logs.go:123] Gathering logs for dmesg ...
I0223 01:23:23.100052 662830 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0223 01:23:23.125369 662830 logs.go:123] Gathering logs for describe nodes ...
I0223 01:23:23.125399 662830 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0223 01:23:23.184890 662830 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0223 01:23:23.184915 662830 logs.go:123] Gathering logs for Docker ...
I0223 01:23:23.184932 662830 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u docker -u cri-docker -n 400"
W0223 01:23:23.205836 662830 out.go:364] Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0223 01:23:23.205882 662830 out.go:239] *
*
W0223 01:23:23.205947 662830 out.go:239] X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0223 01:23:23.205982 662830 out.go:239] *
*
W0223 01:23:23.206941 662830 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0223 01:23:23.209470 662830 out.go:177] X Problems detected in kubelet:
I0223 01:23:23.210676 662830 out.go:177] Feb 23 01:23:02 old-k8s-version-799707 kubelet[5695]: E0223 01:23:02.555197 5695 pod_workers.go:191] Error syncing pod b3d303074fe0ca1d42a8bd9ed248df09 ("kube-scheduler-old-k8s-version-799707_kube-system(b3d303074fe0ca1d42a8bd9ed248df09)"), skipping: failed to "StartContainer" for "kube-scheduler" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-scheduler:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-scheduler:v1.16.0\" is not set"
I0223 01:23:23.212104 662830 out.go:177] Feb 23 01:23:05 old-k8s-version-799707 kubelet[5695]: E0223 01:23:05.555141 5695 pod_workers.go:191] Error syncing pod 9fc427d2e6746d2b3f18846f6f0fcafb ("kube-controller-manager-old-k8s-version-799707_kube-system(9fc427d2e6746d2b3f18846f6f0fcafb)"), skipping: failed to "StartContainer" for "kube-controller-manager" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-controller-manager:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-controller-manager:v1.16.0\" is not set"
I0223 01:23:23.213506 662830 out.go:177] Feb 23 01:23:06 old-k8s-version-799707 kubelet[5695]: E0223 01:23:06.555495 5695 pod_workers.go:191] Error syncing pod ada9a48d69d0428e9c15f5ad9c81ef07 ("etcd-old-k8s-version-799707_kube-system(ada9a48d69d0428e9c15f5ad9c81ef07)"), skipping: failed to "StartContainer" for "etcd" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/etcd:3.3.15-0\": Id or size of image \"k8s.gcr.io/etcd:3.3.15-0\" is not set"
I0223 01:23:23.216196 662830 out.go:177]
W0223 01:23:23.217519 662830 out.go:239] X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0223 01:23:23.217573 662830 out.go:239] * Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
* Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
W0223 01:23:23.217602 662830 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/4172
* Related issue: https://github.com/kubernetes/minikube/issues/4172
I0223 01:23:23.219109 662830 out.go:177]
** /stderr **
start_stop_delete_test.go:188: failed starting minikube -first start-. args "out/minikube-linux-amd64 start -p old-k8s-version-799707 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --container-runtime=docker --kubernetes-version=v1.16.0": exit status 109
helpers_test.go:222: -----------------------post-mortem--------------------------------
helpers_test.go:230: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/FirstStart]: docker inspect <======
helpers_test.go:231: (dbg) Run: docker inspect old-k8s-version-799707
helpers_test.go:235: (dbg) docker inspect old-k8s-version-799707:
-- stdout --
[
{
"Id": "f679df36dcf990835f9af969714a9f7aef6a6f4e8756578784e54dc46c3ccfef",
"Created": "2024-02-23T01:15:05.474444114Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 666050,
"ExitCode": 0,
"Error": "",
"StartedAt": "2024-02-23T01:15:05.794433523Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:78bbddd92c7656e5a7bf9651f59e6b47aa97241efd2e93247c457ec76b2185c5",
"ResolvConfPath": "/var/lib/docker/containers/f679df36dcf990835f9af969714a9f7aef6a6f4e8756578784e54dc46c3ccfef/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/f679df36dcf990835f9af969714a9f7aef6a6f4e8756578784e54dc46c3ccfef/hostname",
"HostsPath": "/var/lib/docker/containers/f679df36dcf990835f9af969714a9f7aef6a6f4e8756578784e54dc46c3ccfef/hosts",
"LogPath": "/var/lib/docker/containers/f679df36dcf990835f9af969714a9f7aef6a6f4e8756578784e54dc46c3ccfef/f679df36dcf990835f9af969714a9f7aef6a6f4e8756578784e54dc46c3ccfef-json.log",
"Name": "/old-k8s-version-799707",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "unconfined",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro",
"old-k8s-version-799707:/var"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {
"max-size": "100m"
}
},
"NetworkMode": "old-k8s-version-799707",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "host",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 2306867200,
"NanoCpus": 2000000000,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": [],
"BlkioDeviceWriteBps": [],
"BlkioDeviceReadIOps": [],
"BlkioDeviceWriteIOps": [],
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 4613734400,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": [],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/e2722ea5301b511ffa3da3e66dbd1d633ae1270718cb4e5e318ff35486007a68-init/diff:/var/lib/docker/overlay2/b6c3064e580e9d3be1c1e7c2f22af1522ce3c491365d231a5e8d9c0e313889c5/diff",
"MergedDir": "/var/lib/docker/overlay2/e2722ea5301b511ffa3da3e66dbd1d633ae1270718cb4e5e318ff35486007a68/merged",
"UpperDir": "/var/lib/docker/overlay2/e2722ea5301b511ffa3da3e66dbd1d633ae1270718cb4e5e318ff35486007a68/diff",
"WorkDir": "/var/lib/docker/overlay2/e2722ea5301b511ffa3da3e66dbd1d633ae1270718cb4e5e318ff35486007a68/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "old-k8s-version-799707",
"Source": "/var/lib/docker/volumes/old-k8s-version-799707/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "old-k8s-version-799707",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"22/tcp": {},
"2376/tcp": {},
"32443/tcp": {},
"5000/tcp": {},
"8443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "old-k8s-version-799707",
"name.minikube.sigs.k8s.io": "old-k8s-version-799707",
"role.minikube.sigs.k8s.io": ""
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "b5c1af4776bf69cc328dfffdf7107f9822a0ea56bd95b5a12ee20bbcffb22663",
"SandboxKey": "/var/run/docker/netns/b5c1af4776bf",
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33358"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33357"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33353"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33355"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33354"
}
]
},
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"old-k8s-version-799707": {
"IPAMConfig": {
"IPv4Address": "192.168.94.2"
},
"Links": null,
"Aliases": [
"f679df36dcf9",
"old-k8s-version-799707"
],
"MacAddress": "02:42:c0:a8:5e:02",
"NetworkID": "bd295bc817aac655859be5f1040d2c41b5d0e7f3be9c06731d2af745450199fa",
"EndpointID": "53bcfb6557fc1d2287655cd3e4d1b7970b1c6c19f1eaa786648cd8dc9931ac6c",
"Gateway": "192.168.94.1",
"IPAddress": "192.168.94.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"DriverOpts": null,
"DNSNames": [
"old-k8s-version-799707",
"f679df36dcf9"
]
}
}
}
}
]
-- /stdout --
helpers_test.go:239: (dbg) Run: out/minikube-linux-amd64 status --format={{.Host}} -p old-k8s-version-799707 -n old-k8s-version-799707
helpers_test.go:239: (dbg) Non-zero exit: out/minikube-linux-amd64 status --format={{.Host}} -p old-k8s-version-799707 -n old-k8s-version-799707: exit status 6 (302.784286ms)
-- stdout --
Running
WARNING: Your kubectl is pointing to stale minikube-vm.
To fix the kubectl context, run `minikube update-context`
-- /stdout --
** stderr **
E0223 01:23:23.581412 760881 status.go:415] kubeconfig endpoint: extract IP: "old-k8s-version-799707" does not appear in /home/jenkins/minikube-integration/18233-317564/kubeconfig
** /stderr **
helpers_test.go:239: status error: exit status 6 (may be ok)
helpers_test.go:241: "old-k8s-version-799707" host is not running, skipping log retrieval (state="Running\nWARNING: Your kubectl is pointing to stale minikube-vm.\nTo fix the kubectl context, run `minikube update-context`")
--- FAIL: TestStartStop/group/old-k8s-version/serial/FirstStart (506.92s)