=== RUN TestStartStop/group/old-k8s-version/serial/FirstStart
start_stop_delete_test.go:186: (dbg) Run: out/minikube-linux-amd64 start -p old-k8s-version-478853 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --container-runtime=docker --kubernetes-version=v1.16.0
start_stop_delete_test.go:186: (dbg) Non-zero exit: out/minikube-linux-amd64 start -p old-k8s-version-478853 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --container-runtime=docker --kubernetes-version=v1.16.0: exit status 109 (8m24.198424507s)
-- stdout --
* [old-k8s-version-478853] minikube v1.32.0 on Ubuntu 20.04 (kvm/amd64)
- MINIKUBE_LOCATION=17936
- MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
- KUBECONFIG=/home/jenkins/minikube-integration/17936-6821/kubeconfig
- MINIKUBE_HOME=/home/jenkins/minikube-integration/17936-6821/.minikube
- MINIKUBE_BIN=out/minikube-linux-amd64
- MINIKUBE_FORCE_SYSTEMD=
* Using the docker driver based on user configuration
* Using Docker driver with root privileges
* Starting control plane node old-k8s-version-478853 in cluster old-k8s-version-478853
* Pulling base image v0.0.42-1708008208-17936 ...
* Creating docker container (CPUs=2, Memory=2200MB) ...
* Preparing Kubernetes v1.16.0 on Docker 25.0.3 ...
- Generating certificates and keys ...
- Booting up control plane ...
- Generating certificates and keys ...
- Booting up control plane ...
X Problems detected in kubelet:
Feb 16 17:36:03 old-k8s-version-478853 kubelet[5716]: E0216 17:36:03.828321 5716 pod_workers.go:191] Error syncing pod 75eac6fd65f4f8477f5572974a7da828 ("etcd-old-k8s-version-478853_kube-system(75eac6fd65f4f8477f5572974a7da828)"), skipping: failed to "StartContainer" for "etcd" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/etcd:3.3.15-0\": Id or size of image \"k8s.gcr.io/etcd:3.3.15-0\" is not set"
Feb 16 17:36:04 old-k8s-version-478853 kubelet[5716]: E0216 17:36:04.833813 5716 pod_workers.go:191] Error syncing pod 9fc427d2e6746d2b3f18846f6f0fcafb ("kube-controller-manager-old-k8s-version-478853_kube-system(9fc427d2e6746d2b3f18846f6f0fcafb)"), skipping: failed to "StartContainer" for "kube-controller-manager" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-controller-manager:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-controller-manager:v1.16.0\" is not set"
Feb 16 17:36:06 old-k8s-version-478853 kubelet[5716]: E0216 17:36:06.833227 5716 pod_workers.go:191] Error syncing pod b3d303074fe0ca1d42a8bd9ed248df09 ("kube-scheduler-old-k8s-version-478853_kube-system(b3d303074fe0ca1d42a8bd9ed248df09)"), skipping: failed to "StartContainer" for "kube-scheduler" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-scheduler:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-scheduler:v1.16.0\" is not set"
-- /stdout --
** stderr **
I0216 17:28:01.011410 354075 out.go:291] Setting OutFile to fd 1 ...
I0216 17:28:01.012556 354075 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0216 17:28:01.012564 354075 out.go:304] Setting ErrFile to fd 2...
I0216 17:28:01.012581 354075 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0216 17:28:01.012935 354075 root.go:338] Updating PATH: /home/jenkins/minikube-integration/17936-6821/.minikube/bin
I0216 17:28:01.013829 354075 out.go:298] Setting JSON to false
I0216 17:28:01.016230 354075 start.go:129] hostinfo: {"hostname":"ubuntu-20-agent","uptime":4227,"bootTime":1708100254,"procs":450,"os":"linux","platform":"ubuntu","platformFamily":"debian","platformVersion":"20.04","kernelVersion":"5.15.0-1051-gcp","kernelArch":"x86_64","virtualizationSystem":"kvm","virtualizationRole":"guest","hostId":"591c9f12-2938-3743-e2bf-c56a050d43d1"}
I0216 17:28:01.016336 354075 start.go:139] virtualization: kvm guest
I0216 17:28:01.018836 354075 out.go:177] * [old-k8s-version-478853] minikube v1.32.0 on Ubuntu 20.04 (kvm/amd64)
I0216 17:28:01.020814 354075 out.go:177] - MINIKUBE_LOCATION=17936
I0216 17:28:01.020849 354075 notify.go:220] Checking for updates...
I0216 17:28:01.022511 354075 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0216 17:28:01.024034 354075 out.go:177] - KUBECONFIG=/home/jenkins/minikube-integration/17936-6821/kubeconfig
I0216 17:28:01.025615 354075 out.go:177] - MINIKUBE_HOME=/home/jenkins/minikube-integration/17936-6821/.minikube
I0216 17:28:01.027106 354075 out.go:177] - MINIKUBE_BIN=out/minikube-linux-amd64
I0216 17:28:01.028490 354075 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0216 17:28:01.030716 354075 config.go:182] Loaded profile config "bridge-123826": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.28.4
I0216 17:28:01.030890 354075 config.go:182] Loaded profile config "kubenet-123826": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.28.4
I0216 17:28:01.031023 354075 config.go:182] Loaded profile config "kubernetes-upgrade-001550": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.16.0
I0216 17:28:01.031144 354075 driver.go:392] Setting default libvirt URI to qemu:///system
I0216 17:28:01.073375 354075 docker.go:122] docker version: linux-25.0.3:Docker Engine - Community
I0216 17:28:01.073476 354075 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0216 17:28:01.163797 354075 info.go:266] docker info: {ID:TS6T:UINC:MIYS:RZPA:KS6T:4JQK:7JHN:D6RA:LDP2:MHAE:G32M:C5NQ Containers:3 ContainersRunning:3 ContainersPaused:0 ContainersStopped:0 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:62 OomKillDisable:true NGoroutines:87 SystemTime:2024-02-16 17:28:01.153733197 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1051-gcp OperatingSystem:Ubuntu 20.04.6 LTS OSType:linux Architecture:x86
_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:8 MemTotal:33648001024 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ubuntu-20-agent Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=builtin] ProductLicense: Warnings:<nil> ServerError
s:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.5] map[Name:scan Path:/usr/libexec/docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.23.0]] Warnings:<nil>}}
I0216 17:28:01.163916 354075 docker.go:295] overlay module found
I0216 17:28:01.165860 354075 out.go:177] * Using the docker driver based on user configuration
I0216 17:28:01.167828 354075 start.go:299] selected driver: docker
I0216 17:28:01.167849 354075 start.go:903] validating driver "docker" against <nil>
I0216 17:28:01.167865 354075 start.go:914] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0216 17:28:01.169071 354075 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0216 17:28:01.247516 354075 info.go:266] docker info: {ID:TS6T:UINC:MIYS:RZPA:KS6T:4JQK:7JHN:D6RA:LDP2:MHAE:G32M:C5NQ Containers:3 ContainersRunning:3 ContainersPaused:0 ContainersStopped:0 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:62 OomKillDisable:true NGoroutines:87 SystemTime:2024-02-16 17:28:01.235339949 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1051-gcp OperatingSystem:Ubuntu 20.04.6 LTS OSType:linux Architecture:x86
_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:8 MemTotal:33648001024 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ubuntu-20-agent Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=builtin] ProductLicense: Warnings:<nil> ServerError
s:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.5] map[Name:scan Path:/usr/libexec/docker/cli-plugins/docker-scan SchemaVersion:0.1.0 ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.23.0]] Warnings:<nil>}}
I0216 17:28:01.247725 354075 start_flags.go:309] no existing cluster config was found, will generate one from the flags
I0216 17:28:01.247968 354075 start_flags.go:931] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0216 17:28:01.250145 354075 out.go:177] * Using Docker driver with root privileges
I0216 17:28:01.251626 354075 cni.go:84] Creating CNI manager for ""
I0216 17:28:01.251658 354075 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0216 17:28:01.251671 354075 start_flags.go:323] config:
{Name:old-k8s-version-478853 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-478853 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local Containe
rRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0216 17:28:01.253389 354075 out.go:177] * Starting control plane node old-k8s-version-478853 in cluster old-k8s-version-478853
I0216 17:28:01.254905 354075 cache.go:121] Beginning downloading kic base image for docker with docker
I0216 17:28:01.256610 354075 out.go:177] * Pulling base image v0.0.42-1708008208-17936 ...
I0216 17:28:01.258317 354075 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0216 17:28:01.258345 354075 image.go:79] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf in local docker daemon
I0216 17:28:01.258364 354075 preload.go:148] Found local preload: /home/jenkins/minikube-integration/17936-6821/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4
I0216 17:28:01.258373 354075 cache.go:56] Caching tarball of preloaded images
I0216 17:28:01.258470 354075 preload.go:174] Found /home/jenkins/minikube-integration/17936-6821/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0216 17:28:01.258478 354075 cache.go:59] Finished verifying existence of preloaded tar for v1.16.0 on docker
I0216 17:28:01.258580 354075 profile.go:148] Saving config to /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/config.json ...
I0216 17:28:01.258598 354075 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/config.json: {Name:mk1ad9eb4deeb05d969d775f06d57fdf99173ad9 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:28:01.277181 354075 image.go:83] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf in local docker daemon, skipping pull
I0216 17:28:01.277211 354075 cache.go:144] gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf exists in daemon, skipping load
I0216 17:28:01.277230 354075 cache.go:194] Successfully downloaded all kic artifacts
I0216 17:28:01.277259 354075 start.go:365] acquiring machines lock for old-k8s-version-478853: {Name:mkde5e52743909de9e75497b3ed0dd80f14fc0ff Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0216 17:28:01.277360 354075 start.go:369] acquired machines lock for "old-k8s-version-478853" in 82.29µs
I0216 17:28:01.277389 354075 start.go:93] Provisioning new machine with config: &{Name:old-k8s-version-478853 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-478853 Namespace:default APIServer
Name:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimiza
tions:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s} &{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}
I0216 17:28:01.277463 354075 start.go:125] createHost starting for "" (driver="docker")
I0216 17:28:01.279701 354075 out.go:204] * Creating docker container (CPUs=2, Memory=2200MB) ...
I0216 17:28:01.279952 354075 start.go:159] libmachine.API.Create for "old-k8s-version-478853" (driver="docker")
I0216 17:28:01.279981 354075 client.go:168] LocalClient.Create starting
I0216 17:28:01.280050 354075 main.go:141] libmachine: Reading certificate data from /home/jenkins/minikube-integration/17936-6821/.minikube/certs/ca.pem
I0216 17:28:01.280080 354075 main.go:141] libmachine: Decoding PEM data...
I0216 17:28:01.280101 354075 main.go:141] libmachine: Parsing certificate...
I0216 17:28:01.280167 354075 main.go:141] libmachine: Reading certificate data from /home/jenkins/minikube-integration/17936-6821/.minikube/certs/cert.pem
I0216 17:28:01.280194 354075 main.go:141] libmachine: Decoding PEM data...
I0216 17:28:01.280211 354075 main.go:141] libmachine: Parsing certificate...
I0216 17:28:01.280541 354075 cli_runner.go:164] Run: docker network inspect old-k8s-version-478853 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
W0216 17:28:01.301192 354075 cli_runner.go:211] docker network inspect old-k8s-version-478853 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1
I0216 17:28:01.301276 354075 network_create.go:281] running [docker network inspect old-k8s-version-478853] to gather additional debugging logs...
I0216 17:28:01.301303 354075 cli_runner.go:164] Run: docker network inspect old-k8s-version-478853
W0216 17:28:01.322261 354075 cli_runner.go:211] docker network inspect old-k8s-version-478853 returned with exit code 1
I0216 17:28:01.322295 354075 network_create.go:284] error running [docker network inspect old-k8s-version-478853]: docker network inspect old-k8s-version-478853: exit status 1
stdout:
[]
stderr:
Error response from daemon: network old-k8s-version-478853 not found
I0216 17:28:01.322319 354075 network_create.go:286] output of [docker network inspect old-k8s-version-478853]: -- stdout --
[]
-- /stdout --
** stderr **
Error response from daemon: network old-k8s-version-478853 not found
** /stderr **
I0216 17:28:01.322429 354075 cli_runner.go:164] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0216 17:28:01.342630 354075 network.go:212] skipping subnet 192.168.49.0/24 that is taken: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName:br-c4eff5c28743 IfaceIPv4:192.168.49.1 IfaceMTU:1500 IfaceMAC:02:42:04:d1:63:57} reservation:<nil>}
I0216 17:28:01.343667 354075 network.go:212] skipping subnet 192.168.58.0/24 that is taken: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 IsPrivate:true Interface:{IfaceName:br-da77939dda2e IfaceIPv4:192.168.58.1 IfaceMTU:1500 IfaceMAC:02:42:54:6a:64:c8} reservation:<nil>}
I0216 17:28:01.344712 354075 network.go:212] skipping subnet 192.168.67.0/24 that is taken: &{IP:192.168.67.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.67.0/24 Gateway:192.168.67.1 ClientMin:192.168.67.2 ClientMax:192.168.67.254 Broadcast:192.168.67.255 IsPrivate:true Interface:{IfaceName:br-05405816cf1b IfaceIPv4:192.168.67.1 IfaceMTU:1500 IfaceMAC:02:42:5b:78:51:34} reservation:<nil>}
I0216 17:28:01.345910 354075 network.go:207] using free private subnet 192.168.76.0/24: &{IP:192.168.76.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.76.0/24 Gateway:192.168.76.1 ClientMin:192.168.76.2 ClientMax:192.168.76.254 Broadcast:192.168.76.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc002933bb0}
I0216 17:28:01.345989 354075 network_create.go:124] attempt to create docker network old-k8s-version-478853 192.168.76.0/24 with gateway 192.168.76.1 and MTU of 1500 ...
I0216 17:28:01.346061 354075 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.76.0/24 --gateway=192.168.76.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=1500 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-478853 old-k8s-version-478853
I0216 17:28:01.426365 354075 network_create.go:108] docker network old-k8s-version-478853 192.168.76.0/24 created
I0216 17:28:01.426400 354075 kic.go:121] calculated static IP "192.168.76.2" for the "old-k8s-version-478853" container
I0216 17:28:01.426478 354075 cli_runner.go:164] Run: docker ps -a --format {{.Names}}
I0216 17:28:01.446138 354075 cli_runner.go:164] Run: docker volume create old-k8s-version-478853 --label name.minikube.sigs.k8s.io=old-k8s-version-478853 --label created_by.minikube.sigs.k8s.io=true
I0216 17:28:01.468025 354075 oci.go:103] Successfully created a docker volume old-k8s-version-478853
I0216 17:28:01.468111 354075 cli_runner.go:164] Run: docker run --rm --name old-k8s-version-478853-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-478853 --entrypoint /usr/bin/test -v old-k8s-version-478853:/var gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf -d /var/lib
I0216 17:28:02.032243 354075 oci.go:107] Successfully prepared a docker volume old-k8s-version-478853
I0216 17:28:02.032288 354075 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0216 17:28:02.032311 354075 kic.go:194] Starting extracting preloaded images to volume ...
I0216 17:28:02.032382 354075 cli_runner.go:164] Run: docker run --rm --entrypoint /usr/bin/tar -v /home/jenkins/minikube-integration/17936-6821/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-478853:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf -I lz4 -xf /preloaded.tar -C /extractDir
I0216 17:28:05.263760 354075 cli_runner.go:217] Completed: docker run --rm --entrypoint /usr/bin/tar -v /home/jenkins/minikube-integration/17936-6821/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-478853:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf -I lz4 -xf /preloaded.tar -C /extractDir: (3.231315763s)
I0216 17:28:05.263797 354075 kic.go:203] duration metric: took 3.231482 seconds to extract preloaded images to volume
W0216 17:28:05.263938 354075 cgroups_linux.go:77] Your kernel does not support swap limit capabilities or the cgroup is not mounted.
I0216 17:28:05.264056 354075 cli_runner.go:164] Run: docker info --format "'{{json .SecurityOptions}}'"
I0216 17:28:05.326810 354075 cli_runner.go:164] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname old-k8s-version-478853 --name old-k8s-version-478853 --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-478853 --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=old-k8s-version-478853 --network old-k8s-version-478853 --ip 192.168.76.2 --volume old-k8s-version-478853:/var --security-opt apparmor=unconfined --memory=2200mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf
I0216 17:28:05.706521 354075 cli_runner.go:164] Run: docker container inspect old-k8s-version-478853 --format={{.State.Running}}
I0216 17:28:05.726476 354075 cli_runner.go:164] Run: docker container inspect old-k8s-version-478853 --format={{.State.Status}}
I0216 17:28:05.747559 354075 cli_runner.go:164] Run: docker exec old-k8s-version-478853 stat /var/lib/dpkg/alternatives/iptables
I0216 17:28:05.792639 354075 oci.go:144] the created container "old-k8s-version-478853" has a running status.
I0216 17:28:05.792677 354075 kic.go:225] Creating ssh key for kic: /home/jenkins/minikube-integration/17936-6821/.minikube/machines/old-k8s-version-478853/id_rsa...
I0216 17:28:05.975348 354075 kic_runner.go:191] docker (temp): /home/jenkins/minikube-integration/17936-6821/.minikube/machines/old-k8s-version-478853/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0216 17:28:06.004626 354075 cli_runner.go:164] Run: docker container inspect old-k8s-version-478853 --format={{.State.Status}}
I0216 17:28:06.025303 354075 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0216 17:28:06.025340 354075 kic_runner.go:114] Args: [docker exec --privileged old-k8s-version-478853 chown docker:docker /home/docker/.ssh/authorized_keys]
I0216 17:28:06.078615 354075 cli_runner.go:164] Run: docker container inspect old-k8s-version-478853 --format={{.State.Status}}
I0216 17:28:06.102241 354075 machine.go:88] provisioning docker machine ...
I0216 17:28:06.102284 354075 ubuntu.go:169] provisioning hostname "old-k8s-version-478853"
I0216 17:28:06.102350 354075 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-478853
I0216 17:28:06.130820 354075 main.go:141] libmachine: Using SSH client type: native
I0216 17:28:06.131376 354075 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x80a4a0] 0x80d180 <nil> [] 0s} 127.0.0.1 33052 <nil> <nil>}
I0216 17:28:06.131403 354075 main.go:141] libmachine: About to run SSH command:
sudo hostname old-k8s-version-478853 && echo "old-k8s-version-478853" | sudo tee /etc/hostname
I0216 17:28:06.132171 354075 main.go:141] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:36092->127.0.0.1:33052: read: connection reset by peer
I0216 17:28:09.285249 354075 main.go:141] libmachine: SSH cmd err, output: <nil>: old-k8s-version-478853
I0216 17:28:09.285407 354075 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-478853
I0216 17:28:09.306312 354075 main.go:141] libmachine: Using SSH client type: native
I0216 17:28:09.306715 354075 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x80a4a0] 0x80d180 <nil> [] 0s} 127.0.0.1 33052 <nil> <nil>}
I0216 17:28:09.306743 354075 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sold-k8s-version-478853' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 old-k8s-version-478853/g' /etc/hosts;
else
echo '127.0.1.1 old-k8s-version-478853' | sudo tee -a /etc/hosts;
fi
fi
I0216 17:28:09.444951 354075 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0216 17:28:09.444991 354075 ubuntu.go:175] set auth options {CertDir:/home/jenkins/minikube-integration/17936-6821/.minikube CaCertPath:/home/jenkins/minikube-integration/17936-6821/.minikube/certs/ca.pem CaPrivateKeyPath:/home/jenkins/minikube-integration/17936-6821/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/home/jenkins/minikube-integration/17936-6821/.minikube/machines/server.pem ServerKeyPath:/home/jenkins/minikube-integration/17936-6821/.minikube/machines/server-key.pem ClientKeyPath:/home/jenkins/minikube-integration/17936-6821/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/home/jenkins/minikube-integration/17936-6821/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/home/jenkins/minikube-integration/17936-6821/.minikube}
I0216 17:28:09.445031 354075 ubuntu.go:177] setting up certificates
I0216 17:28:09.445053 354075 provision.go:83] configureAuth start
I0216 17:28:09.445130 354075 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-478853
I0216 17:28:09.463005 354075 provision.go:138] copyHostCerts
I0216 17:28:09.463076 354075 exec_runner.go:144] found /home/jenkins/minikube-integration/17936-6821/.minikube/key.pem, removing ...
I0216 17:28:09.463092 354075 exec_runner.go:203] rm: /home/jenkins/minikube-integration/17936-6821/.minikube/key.pem
I0216 17:28:09.463185 354075 exec_runner.go:151] cp: /home/jenkins/minikube-integration/17936-6821/.minikube/certs/key.pem --> /home/jenkins/minikube-integration/17936-6821/.minikube/key.pem (1679 bytes)
I0216 17:28:09.463334 354075 exec_runner.go:144] found /home/jenkins/minikube-integration/17936-6821/.minikube/ca.pem, removing ...
I0216 17:28:09.463351 354075 exec_runner.go:203] rm: /home/jenkins/minikube-integration/17936-6821/.minikube/ca.pem
I0216 17:28:09.463394 354075 exec_runner.go:151] cp: /home/jenkins/minikube-integration/17936-6821/.minikube/certs/ca.pem --> /home/jenkins/minikube-integration/17936-6821/.minikube/ca.pem (1082 bytes)
I0216 17:28:09.463479 354075 exec_runner.go:144] found /home/jenkins/minikube-integration/17936-6821/.minikube/cert.pem, removing ...
I0216 17:28:09.463492 354075 exec_runner.go:203] rm: /home/jenkins/minikube-integration/17936-6821/.minikube/cert.pem
I0216 17:28:09.463527 354075 exec_runner.go:151] cp: /home/jenkins/minikube-integration/17936-6821/.minikube/certs/cert.pem --> /home/jenkins/minikube-integration/17936-6821/.minikube/cert.pem (1123 bytes)
I0216 17:28:09.463604 354075 provision.go:112] generating server cert: /home/jenkins/minikube-integration/17936-6821/.minikube/machines/server.pem ca-key=/home/jenkins/minikube-integration/17936-6821/.minikube/certs/ca.pem private-key=/home/jenkins/minikube-integration/17936-6821/.minikube/certs/ca-key.pem org=jenkins.old-k8s-version-478853 san=[192.168.76.2 127.0.0.1 localhost 127.0.0.1 minikube old-k8s-version-478853]
I0216 17:28:09.615979 354075 provision.go:172] copyRemoteCerts
I0216 17:28:09.616038 354075 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0216 17:28:09.616071 354075 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-478853
I0216 17:28:09.633687 354075 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33052 SSHKeyPath:/home/jenkins/minikube-integration/17936-6821/.minikube/machines/old-k8s-version-478853/id_rsa Username:docker}
I0216 17:28:09.730017 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1082 bytes)
I0216 17:28:09.753281 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/machines/server.pem --> /etc/docker/server.pem (1241 bytes)
I0216 17:28:09.776282 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1675 bytes)
I0216 17:28:09.798788 354075 provision.go:86] duration metric: configureAuth took 353.722077ms
I0216 17:28:09.798821 354075 ubuntu.go:193] setting minikube options for container-runtime
I0216 17:28:09.799019 354075 config.go:182] Loaded profile config "old-k8s-version-478853": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.16.0
I0216 17:28:09.799087 354075 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-478853
I0216 17:28:09.817140 354075 main.go:141] libmachine: Using SSH client type: native
I0216 17:28:09.817673 354075 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x80a4a0] 0x80d180 <nil> [] 0s} 127.0.0.1 33052 <nil> <nil>}
I0216 17:28:09.817706 354075 main.go:141] libmachine: About to run SSH command:
df --output=fstype / | tail -n 1
I0216 17:28:09.948612 354075 main.go:141] libmachine: SSH cmd err, output: <nil>: overlay
I0216 17:28:09.948640 354075 ubuntu.go:71] root file system type: overlay
I0216 17:28:09.948807 354075 provision.go:309] Updating docker unit: /lib/systemd/system/docker.service ...
I0216 17:28:09.948880 354075 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-478853
I0216 17:28:09.967168 354075 main.go:141] libmachine: Using SSH client type: native
I0216 17:28:09.967526 354075 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x80a4a0] 0x80d180 <nil> [] 0s} 127.0.0.1 33052 <nil> <nil>}
I0216 17:28:09.967592 354075 main.go:141] libmachine: About to run SSH command:
sudo mkdir -p /lib/systemd/system && printf %s "[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP \$MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service.new
I0216 17:28:10.111358 354075 main.go:141] libmachine: SSH cmd err, output: <nil>: [Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
I0216 17:28:10.111444 354075 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-478853
I0216 17:28:10.128916 354075 main.go:141] libmachine: Using SSH client type: native
I0216 17:28:10.129289 354075 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x80a4a0] 0x80d180 <nil> [] 0s} 127.0.0.1 33052 <nil> <nil>}
I0216 17:28:10.129315 354075 main.go:141] libmachine: About to run SSH command:
sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
I0216 17:28:10.834438 354075 main.go:141] libmachine: SSH cmd err, output: <nil>: --- /lib/systemd/system/docker.service 2024-02-06 21:12:51.000000000 +0000
+++ /lib/systemd/system/docker.service.new 2024-02-16 17:28:10.106751193 +0000
@@ -1,46 +1,49 @@
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
-After=network-online.target docker.socket firewalld.service containerd.service time-set.target
-Wants=network-online.target containerd.service
+BindsTo=containerd.service
+After=network-online.target firewalld.service containerd.service
+Wants=network-online.target
Requires=docker.socket
+StartLimitBurst=3
+StartLimitIntervalSec=60
[Service]
Type=notify
-# the default is not to use systemd for cgroups because the delegate issues still
-# exists and systemd currently does not support the cgroup feature set required
-# for containers run by docker
-ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
-ExecReload=/bin/kill -s HUP $MAINPID
-TimeoutStartSec=0
-RestartSec=2
-Restart=always
+Restart=on-failure
-# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
-# Both the old, and new location are accepted by systemd 229 and up, so using the old location
-# to make them work for either version of systemd.
-StartLimitBurst=3
-# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
-# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
-# this option work for either version of systemd.
-StartLimitInterval=60s
+
+# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
+# The base configuration already specifies an 'ExecStart=...' command. The first directive
+# here is to clear out that command inherited from the base configuration. Without this,
+# the command from the base configuration and the command specified here are treated as
+# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
+# will catch this invalid input and refuse to start the service with an error like:
+# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
+
+# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
+# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
+ExecStart=
+ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
+ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
+LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
-# Comment TasksMax if your systemd version does not support it.
-# Only systemd 226 and above support this option.
+# Uncomment TasksMax if your systemd version supports it.
+# Only systemd 226 and above support this version.
TasksMax=infinity
+TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
-OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
I0216 17:28:10.834474 354075 machine.go:91] provisioned docker machine in 4.73220419s
I0216 17:28:10.834486 354075 client.go:171] LocalClient.Create took 9.554499734s
I0216 17:28:10.834507 354075 start.go:167] duration metric: libmachine.API.Create for "old-k8s-version-478853" took 9.554554941s
I0216 17:28:10.834517 354075 start.go:300] post-start starting for "old-k8s-version-478853" (driver="docker")
I0216 17:28:10.834530 354075 start.go:329] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0216 17:28:10.834592 354075 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0216 17:28:10.834635 354075 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-478853
I0216 17:28:10.851790 354075 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33052 SSHKeyPath:/home/jenkins/minikube-integration/17936-6821/.minikube/machines/old-k8s-version-478853/id_rsa Username:docker}
I0216 17:28:10.945524 354075 ssh_runner.go:195] Run: cat /etc/os-release
I0216 17:28:10.948721 354075 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0216 17:28:10.948751 354075 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0216 17:28:10.948759 354075 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0216 17:28:10.948766 354075 info.go:137] Remote host: Ubuntu 22.04.3 LTS
I0216 17:28:10.948775 354075 filesync.go:126] Scanning /home/jenkins/minikube-integration/17936-6821/.minikube/addons for local assets ...
I0216 17:28:10.948827 354075 filesync.go:126] Scanning /home/jenkins/minikube-integration/17936-6821/.minikube/files for local assets ...
I0216 17:28:10.948898 354075 filesync.go:149] local asset: /home/jenkins/minikube-integration/17936-6821/.minikube/files/etc/ssl/certs/136192.pem -> 136192.pem in /etc/ssl/certs
I0216 17:28:10.948980 354075 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0216 17:28:10.957100 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/files/etc/ssl/certs/136192.pem --> /etc/ssl/certs/136192.pem (1708 bytes)
I0216 17:28:10.978674 354075 start.go:303] post-start completed in 144.144874ms
I0216 17:28:10.979011 354075 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-478853
I0216 17:28:10.996978 354075 profile.go:148] Saving config to /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/config.json ...
I0216 17:28:10.997256 354075 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0216 17:28:10.997301 354075 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-478853
I0216 17:28:11.015848 354075 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33052 SSHKeyPath:/home/jenkins/minikube-integration/17936-6821/.minikube/machines/old-k8s-version-478853/id_rsa Username:docker}
I0216 17:28:11.105027 354075 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0216 17:28:11.109394 354075 start.go:128] duration metric: createHost completed in 9.831914815s
I0216 17:28:11.109423 354075 start.go:83] releasing machines lock for "old-k8s-version-478853", held for 9.832051827s
I0216 17:28:11.109490 354075 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-478853
I0216 17:28:11.126920 354075 ssh_runner.go:195] Run: cat /version.json
I0216 17:28:11.126967 354075 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-478853
I0216 17:28:11.127026 354075 ssh_runner.go:195] Run: curl -sS -m 2 https://registry.k8s.io/
I0216 17:28:11.127099 354075 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-478853
I0216 17:28:11.145367 354075 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33052 SSHKeyPath:/home/jenkins/minikube-integration/17936-6821/.minikube/machines/old-k8s-version-478853/id_rsa Username:docker}
I0216 17:28:11.145408 354075 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33052 SSHKeyPath:/home/jenkins/minikube-integration/17936-6821/.minikube/machines/old-k8s-version-478853/id_rsa Username:docker}
I0216 17:28:11.235799 354075 ssh_runner.go:195] Run: systemctl --version
I0216 17:28:11.326547 354075 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
I0216 17:28:11.331384 354075 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *loopback.conf* -not -name *.mk_disabled -exec sh -c "grep -q loopback {} && ( grep -q name {} || sudo sed -i '/"type": "loopback"/i \ \ \ \ "name": "loopback",' {} ) && sudo sed -i 's|"cniVersion": ".*"|"cniVersion": "1.0.0"|g' {}" ;
I0216 17:28:11.355367 354075 cni.go:230] loopback cni configuration patched: "/etc/cni/net.d/*loopback.conf*" found
I0216 17:28:11.355451 354075 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *bridge* -not -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e '/"dst": ".*:.*"/d' -e 's|^(.*)"dst": (.*)[,*]$|\1"dst": \2|g' -e '/"subnet": ".*:.*"/d' -e 's|^(.*)"subnet": ".*"(.*)[,*]$|\1"subnet": "10.244.0.0/16"\2|g' {}" ;
I0216 17:28:11.371591 354075 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e 's|^(.*)"subnet": ".*"(.*)$|\1"subnet": "10.244.0.0/16"\2|g' -e 's|^(.*)"gateway": ".*"(.*)$|\1"gateway": "10.244.0.1"\2|g' {}" ;
I0216 17:28:11.387881 354075 cni.go:308] configured [/etc/cni/net.d/100-crio-bridge.conf, /etc/cni/net.d/87-podman-bridge.conflist] bridge cni config(s)
I0216 17:28:11.387911 354075 start.go:475] detecting cgroup driver to use...
I0216 17:28:11.387944 354075 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0216 17:28:11.388060 354075 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0216 17:28:11.404348 354075 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.1"|' /etc/containerd/config.toml"
I0216 17:28:11.413528 354075 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I0216 17:28:11.422614 354075 containerd.go:146] configuring containerd to use "cgroupfs" as cgroup driver...
I0216 17:28:11.422679 354075 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = false|g' /etc/containerd/config.toml"
I0216 17:28:11.431472 354075 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0216 17:28:11.440310 354075 ssh_runner.go:195] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I0216 17:28:11.448955 354075 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0216 17:28:11.457681 354075 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0216 17:28:11.465746 354075 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I0216 17:28:11.474698 354075 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0216 17:28:11.481913 354075 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0216 17:28:11.489035 354075 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0216 17:28:11.575688 354075 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0216 17:28:11.673847 354075 start.go:475] detecting cgroup driver to use...
I0216 17:28:11.673900 354075 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0216 17:28:11.673953 354075 ssh_runner.go:195] Run: sudo systemctl cat docker.service
I0216 17:28:11.686614 354075 cruntime.go:279] skipping containerd shutdown because we are bound to it
I0216 17:28:11.686716 354075 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0216 17:28:11.699498 354075 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///var/run/dockershim.sock
" | sudo tee /etc/crictl.yaml"
I0216 17:28:11.717251 354075 ssh_runner.go:195] Run: which cri-dockerd
I0216 17:28:11.720809 354075 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/cri-docker.service.d
I0216 17:28:11.729549 354075 ssh_runner.go:362] scp memory --> /etc/systemd/system/cri-docker.service.d/10-cni.conf (189 bytes)
I0216 17:28:11.746817 354075 ssh_runner.go:195] Run: sudo systemctl unmask docker.service
I0216 17:28:11.843006 354075 ssh_runner.go:195] Run: sudo systemctl enable docker.socket
I0216 17:28:11.938476 354075 docker.go:574] configuring docker to use "cgroupfs" as cgroup driver...
I0216 17:28:11.938623 354075 ssh_runner.go:362] scp memory --> /etc/docker/daemon.json (130 bytes)
I0216 17:28:11.956099 354075 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0216 17:28:12.036315 354075 ssh_runner.go:195] Run: sudo systemctl restart docker
I0216 17:28:12.311329 354075 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0216 17:28:12.337058 354075 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0216 17:28:12.362639 354075 out.go:204] * Preparing Kubernetes v1.16.0 on Docker 25.0.3 ...
I0216 17:28:12.362741 354075 cli_runner.go:164] Run: docker network inspect old-k8s-version-478853 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0216 17:28:12.381052 354075 ssh_runner.go:195] Run: grep 192.168.76.1 host.minikube.internal$ /etc/hosts
I0216 17:28:12.385275 354075 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.76.1 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0216 17:28:12.395833 354075 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0216 17:28:12.395901 354075 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0216 17:28:12.413961 354075 docker.go:685] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0216 17:28:12.413981 354075 docker.go:691] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0216 17:28:12.414020 354075 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0216 17:28:12.422616 354075 ssh_runner.go:195] Run: which lz4
I0216 17:28:12.426100 354075 ssh_runner.go:195] Run: stat -c "%s %y" /preloaded.tar.lz4
I0216 17:28:12.429269 354075 ssh_runner.go:352] existence check for /preloaded.tar.lz4: stat -c "%s %y" /preloaded.tar.lz4: Process exited with status 1
stdout:
stderr:
stat: cannot statx '/preloaded.tar.lz4': No such file or directory
I0216 17:28:12.429297 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-amd64.tar.lz4 --> /preloaded.tar.lz4 (369789069 bytes)
I0216 17:28:13.459945 354075 docker.go:649] Took 1.033867 seconds to copy over tarball
I0216 17:28:13.460006 354075 ssh_runner.go:195] Run: sudo tar --xattrs --xattrs-include security.capability -I lz4 -C /var -xf /preloaded.tar.lz4
I0216 17:28:16.062052 354075 ssh_runner.go:235] Completed: sudo tar --xattrs --xattrs-include security.capability -I lz4 -C /var -xf /preloaded.tar.lz4: (2.602007284s)
I0216 17:28:16.062126 354075 ssh_runner.go:146] rm: /preloaded.tar.lz4
I0216 17:28:16.124290 354075 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0216 17:28:16.133354 354075 ssh_runner.go:362] scp memory --> /var/lib/docker/image/overlay2/repositories.json (2499 bytes)
I0216 17:28:16.150485 354075 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0216 17:28:16.243378 354075 ssh_runner.go:195] Run: sudo systemctl restart docker
I0216 17:28:18.190622 354075 ssh_runner.go:235] Completed: sudo systemctl restart docker: (1.947201235s)
I0216 17:28:18.190796 354075 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0216 17:28:18.213587 354075 docker.go:685] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0216 17:28:18.213610 354075 docker.go:691] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0216 17:28:18.213619 354075 cache_images.go:88] LoadImages start: [registry.k8s.io/kube-apiserver:v1.16.0 registry.k8s.io/kube-controller-manager:v1.16.0 registry.k8s.io/kube-scheduler:v1.16.0 registry.k8s.io/kube-proxy:v1.16.0 registry.k8s.io/pause:3.1 registry.k8s.io/etcd:3.3.15-0 registry.k8s.io/coredns:1.6.2 gcr.io/k8s-minikube/storage-provisioner:v5]
I0216 17:28:18.215273 354075 image.go:134] retrieving image: gcr.io/k8s-minikube/storage-provisioner:v5
I0216 17:28:18.215481 354075 image.go:134] retrieving image: registry.k8s.io/kube-apiserver:v1.16.0
I0216 17:28:18.215537 354075 image.go:134] retrieving image: registry.k8s.io/kube-controller-manager:v1.16.0
I0216 17:28:18.215678 354075 image.go:134] retrieving image: registry.k8s.io/etcd:3.3.15-0
I0216 17:28:18.215684 354075 image.go:134] retrieving image: registry.k8s.io/pause:3.1
I0216 17:28:18.215741 354075 image.go:134] retrieving image: registry.k8s.io/coredns:1.6.2
I0216 17:28:18.215827 354075 image.go:134] retrieving image: registry.k8s.io/kube-scheduler:v1.16.0
I0216 17:28:18.215851 354075 image.go:134] retrieving image: registry.k8s.io/kube-proxy:v1.16.0
I0216 17:28:18.216033 354075 image.go:177] daemon lookup for gcr.io/k8s-minikube/storage-provisioner:v5: Error response from daemon: No such image: gcr.io/k8s-minikube/storage-provisioner:v5
I0216 17:28:18.216215 354075 image.go:177] daemon lookup for registry.k8s.io/kube-apiserver:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-apiserver:v1.16.0
I0216 17:28:18.216239 354075 image.go:177] daemon lookup for registry.k8s.io/etcd:3.3.15-0: Error response from daemon: No such image: registry.k8s.io/etcd:3.3.15-0
I0216 17:28:18.216216 354075 image.go:177] daemon lookup for registry.k8s.io/kube-controller-manager:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-controller-manager:v1.16.0
I0216 17:28:18.216392 354075 image.go:177] daemon lookup for registry.k8s.io/coredns:1.6.2: Error response from daemon: No such image: registry.k8s.io/coredns:1.6.2
I0216 17:28:18.216439 354075 image.go:177] daemon lookup for registry.k8s.io/pause:3.1: Error response from daemon: No such image: registry.k8s.io/pause:3.1
I0216 17:28:18.216502 354075 image.go:177] daemon lookup for registry.k8s.io/kube-scheduler:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-scheduler:v1.16.0
I0216 17:28:18.216558 354075 image.go:177] daemon lookup for registry.k8s.io/kube-proxy:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-proxy:v1.16.0
I0216 17:28:18.389616 354075 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/pause:3.1
I0216 17:28:18.408313 354075 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-scheduler:v1.16.0
I0216 17:28:18.411378 354075 cache_images.go:116] "registry.k8s.io/pause:3.1" needs transfer: "registry.k8s.io/pause:3.1" does not exist at hash "da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e" in container runtime
I0216 17:28:18.411429 354075 docker.go:337] Removing image: registry.k8s.io/pause:3.1
I0216 17:28:18.411470 354075 ssh_runner.go:195] Run: docker rmi registry.k8s.io/pause:3.1
I0216 17:28:18.417719 354075 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/etcd:3.3.15-0
I0216 17:28:18.430497 354075 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/coredns:1.6.2
I0216 17:28:18.431364 354075 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-controller-manager:v1.16.0
I0216 17:28:18.432824 354075 cache_images.go:116] "registry.k8s.io/kube-scheduler:v1.16.0" needs transfer: "registry.k8s.io/kube-scheduler:v1.16.0" does not exist at hash "301ddc62b80b16315d3c2653cf3888370394277afb3187614cfa20edc352ca0a" in container runtime
I0216 17:28:18.432904 354075 docker.go:337] Removing image: registry.k8s.io/kube-scheduler:v1.16.0
I0216 17:28:18.432983 354075 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-scheduler:v1.16.0
I0216 17:28:18.434185 354075 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/17936-6821/.minikube/cache/images/amd64/registry.k8s.io/pause_3.1
I0216 17:28:18.443205 354075 cache_images.go:116] "registry.k8s.io/etcd:3.3.15-0" needs transfer: "registry.k8s.io/etcd:3.3.15-0" does not exist at hash "b2756210eeabf84f3221da9959e9483f3919dc2aaab4cd45e7cd072fcbde27ed" in container runtime
I0216 17:28:18.443260 354075 docker.go:337] Removing image: registry.k8s.io/etcd:3.3.15-0
I0216 17:28:18.443311 354075 ssh_runner.go:195] Run: docker rmi registry.k8s.io/etcd:3.3.15-0
I0216 17:28:18.455129 354075 cache_images.go:116] "registry.k8s.io/coredns:1.6.2" needs transfer: "registry.k8s.io/coredns:1.6.2" does not exist at hash "bf261d157914477ee1a5969d28ec687f3fbfc9fb5a664b22df78e57023b0e03b" in container runtime
I0216 17:28:18.455194 354075 docker.go:337] Removing image: registry.k8s.io/coredns:1.6.2
I0216 17:28:18.455247 354075 ssh_runner.go:195] Run: docker rmi registry.k8s.io/coredns:1.6.2
I0216 17:28:18.456447 354075 cache_images.go:116] "registry.k8s.io/kube-controller-manager:v1.16.0" needs transfer: "registry.k8s.io/kube-controller-manager:v1.16.0" does not exist at hash "06a629a7e51cdcc81a5ed6a3e6650348312f20c954ac52ee489a023628ec9c7d" in container runtime
I0216 17:28:18.456529 354075 docker.go:337] Removing image: registry.k8s.io/kube-controller-manager:v1.16.0
I0216 17:28:18.456583 354075 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-controller-manager:v1.16.0
I0216 17:28:18.456685 354075 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/17936-6821/.minikube/cache/images/amd64/registry.k8s.io/kube-scheduler_v1.16.0
I0216 17:28:18.501630 354075 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/17936-6821/.minikube/cache/images/amd64/registry.k8s.io/etcd_3.3.15-0
I0216 17:28:18.511511 354075 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/17936-6821/.minikube/cache/images/amd64/registry.k8s.io/coredns_1.6.2
I0216 17:28:18.511594 354075 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/17936-6821/.minikube/cache/images/amd64/registry.k8s.io/kube-controller-manager_v1.16.0
I0216 17:28:18.563741 354075 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-proxy:v1.16.0
I0216 17:28:18.583408 354075 cache_images.go:116] "registry.k8s.io/kube-proxy:v1.16.0" needs transfer: "registry.k8s.io/kube-proxy:v1.16.0" does not exist at hash "c21b0c7400f988db4777858edd13b6d3930d62d7ccf026d2415485a52037f384" in container runtime
I0216 17:28:18.583464 354075 docker.go:337] Removing image: registry.k8s.io/kube-proxy:v1.16.0
I0216 17:28:18.583513 354075 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-proxy:v1.16.0
I0216 17:28:18.604187 354075 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/17936-6821/.minikube/cache/images/amd64/registry.k8s.io/kube-proxy_v1.16.0
I0216 17:28:18.620479 354075 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-apiserver:v1.16.0
I0216 17:28:18.639878 354075 cache_images.go:116] "registry.k8s.io/kube-apiserver:v1.16.0" needs transfer: "registry.k8s.io/kube-apiserver:v1.16.0" does not exist at hash "b305571ca60a5a7818bda47da122683d75e8a1907475681ee8b1efbd06bff12e" in container runtime
I0216 17:28:18.639932 354075 docker.go:337] Removing image: registry.k8s.io/kube-apiserver:v1.16.0
I0216 17:28:18.639979 354075 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-apiserver:v1.16.0
I0216 17:28:18.659922 354075 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/17936-6821/.minikube/cache/images/amd64/registry.k8s.io/kube-apiserver_v1.16.0
I0216 17:28:19.006502 354075 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} gcr.io/k8s-minikube/storage-provisioner:v5
I0216 17:28:19.027017 354075 cache_images.go:92] LoadImages completed in 813.381295ms
W0216 17:28:19.027164 354075 out.go:239] X Unable to load cached images: loading cached images: stat /home/jenkins/minikube-integration/17936-6821/.minikube/cache/images/amd64/registry.k8s.io/pause_3.1: no such file or directory
X Unable to load cached images: loading cached images: stat /home/jenkins/minikube-integration/17936-6821/.minikube/cache/images/amd64/registry.k8s.io/pause_3.1: no such file or directory
I0216 17:28:19.027333 354075 ssh_runner.go:195] Run: docker info --format {{.CgroupDriver}}
I0216 17:28:19.080531 354075 cni.go:84] Creating CNI manager for ""
I0216 17:28:19.080560 354075 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0216 17:28:19.080579 354075 kubeadm.go:87] Using pod CIDR: 10.244.0.0/16
I0216 17:28:19.080602 354075 kubeadm.go:176] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.76.2 APIServerPort:8443 KubernetesVersion:v1.16.0 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:old-k8s-version-478853 NodeName:old-k8s-version-478853 DNSDomain:cluster.local CRISocket:/var/run/dockershim.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.76.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.76.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticP
odPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[hairpinMode:hairpin-veth runtimeRequestTimeout:15m] PrependCriSocketUnix:false}
I0216 17:28:19.080735 354075 kubeadm.go:181] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.76.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: "old-k8s-version-478853"
kubeletExtraArgs:
node-ip: 192.168.76.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.76.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: old-k8s-version-478853
controlPlaneEndpoint: control-plane.minikube.internal:8443
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
listen-metrics-urls: http://127.0.0.1:2381,http://192.168.76.2:2381
kubernetesVersion: v1.16.0
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0216 17:28:19.080801 354075 kubeadm.go:976] kubelet [Unit]
Wants=docker.socket
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.16.0/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=old-k8s-version-478853 --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.76.2
[Install]
config:
{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-478853 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:}
I0216 17:28:19.080845 354075 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.16.0
I0216 17:28:19.089360 354075 binaries.go:44] Found k8s binaries, skipping transfer
I0216 17:28:19.089416 354075 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0216 17:28:19.097743 354075 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (348 bytes)
I0216 17:28:19.115094 354075 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0216 17:28:19.134475 354075 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2174 bytes)
I0216 17:28:19.153896 354075 ssh_runner.go:195] Run: grep 192.168.76.2 control-plane.minikube.internal$ /etc/hosts
I0216 17:28:19.157246 354075 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.76.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0216 17:28:19.171130 354075 certs.go:56] Setting up /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853 for IP: 192.168.76.2
I0216 17:28:19.171166 354075 certs.go:190] acquiring lock for shared ca certs: {Name:mk9d742a64083da672505a071544cb22b9fe542d Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:28:19.171327 354075 certs.go:199] skipping minikubeCA CA generation: /home/jenkins/minikube-integration/17936-6821/.minikube/ca.key
I0216 17:28:19.171383 354075 certs.go:199] skipping proxyClientCA CA generation: /home/jenkins/minikube-integration/17936-6821/.minikube/proxy-client-ca.key
I0216 17:28:19.171439 354075 certs.go:319] generating minikube-user signed cert: /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/client.key
I0216 17:28:19.171452 354075 crypto.go:68] Generating cert /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/client.crt with IP's: []
I0216 17:28:19.290211 354075 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/client.crt ...
I0216 17:28:19.290241 354075 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/client.crt: {Name:mk7f62e86590b8c679337b3895444ae16a224c04 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:28:19.290411 354075 crypto.go:164] Writing key to /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/client.key ...
I0216 17:28:19.290428 354075 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/client.key: {Name:mka667bb270af222937e6ce60939ffdb207e7e7b Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:28:19.290525 354075 certs.go:319] generating minikube signed cert: /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/apiserver.key.31bdca25
I0216 17:28:19.290544 354075 crypto.go:68] Generating cert /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/apiserver.crt.31bdca25 with IP's: [192.168.76.2 10.96.0.1 127.0.0.1 10.0.0.1]
I0216 17:28:19.383987 354075 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/apiserver.crt.31bdca25 ...
I0216 17:28:19.384020 354075 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/apiserver.crt.31bdca25: {Name:mk53eb1fd312e073250b7fe76ef4b7bd700c74ee Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:28:19.384207 354075 crypto.go:164] Writing key to /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/apiserver.key.31bdca25 ...
I0216 17:28:19.384235 354075 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/apiserver.key.31bdca25: {Name:mk89de4481d7a32f70f330f3ed2d95c19f39fc52 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:28:19.384327 354075 certs.go:337] copying /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/apiserver.crt.31bdca25 -> /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/apiserver.crt
I0216 17:28:19.384391 354075 certs.go:341] copying /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/apiserver.key.31bdca25 -> /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/apiserver.key
I0216 17:28:19.384438 354075 certs.go:319] generating aggregator signed cert: /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/proxy-client.key
I0216 17:28:19.384452 354075 crypto.go:68] Generating cert /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/proxy-client.crt with IP's: []
I0216 17:28:19.491703 354075 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/proxy-client.crt ...
I0216 17:28:19.491734 354075 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/proxy-client.crt: {Name:mk28d5d76846b3a8f5a310c7e9c469e3fa2f3a80 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:28:19.491906 354075 crypto.go:164] Writing key to /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/proxy-client.key ...
I0216 17:28:19.491924 354075 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/proxy-client.key: {Name:mk5e160cbceb7333532b8d009d56bd5243cb7d3b Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:28:19.492108 354075 certs.go:437] found cert: /home/jenkins/minikube-integration/17936-6821/.minikube/certs/home/jenkins/minikube-integration/17936-6821/.minikube/certs/13619.pem (1338 bytes)
W0216 17:28:19.492176 354075 certs.go:433] ignoring /home/jenkins/minikube-integration/17936-6821/.minikube/certs/home/jenkins/minikube-integration/17936-6821/.minikube/certs/13619_empty.pem, impossibly tiny 0 bytes
I0216 17:28:19.492209 354075 certs.go:437] found cert: /home/jenkins/minikube-integration/17936-6821/.minikube/certs/home/jenkins/minikube-integration/17936-6821/.minikube/certs/ca-key.pem (1675 bytes)
I0216 17:28:19.492247 354075 certs.go:437] found cert: /home/jenkins/minikube-integration/17936-6821/.minikube/certs/home/jenkins/minikube-integration/17936-6821/.minikube/certs/ca.pem (1082 bytes)
I0216 17:28:19.492289 354075 certs.go:437] found cert: /home/jenkins/minikube-integration/17936-6821/.minikube/certs/home/jenkins/minikube-integration/17936-6821/.minikube/certs/cert.pem (1123 bytes)
I0216 17:28:19.492324 354075 certs.go:437] found cert: /home/jenkins/minikube-integration/17936-6821/.minikube/certs/home/jenkins/minikube-integration/17936-6821/.minikube/certs/key.pem (1679 bytes)
I0216 17:28:19.492395 354075 certs.go:437] found cert: /home/jenkins/minikube-integration/17936-6821/.minikube/files/etc/ssl/certs/home/jenkins/minikube-integration/17936-6821/.minikube/files/etc/ssl/certs/136192.pem (1708 bytes)
I0216 17:28:19.493026 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1399 bytes)
I0216 17:28:19.518158 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1679 bytes)
I0216 17:28:19.542084 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0216 17:28:19.566805 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/profiles/old-k8s-version-478853/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1679 bytes)
I0216 17:28:19.594133 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0216 17:28:19.619793 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1675 bytes)
I0216 17:28:19.652833 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0216 17:28:19.682739 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1675 bytes)
I0216 17:28:19.707405 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/certs/13619.pem --> /usr/share/ca-certificates/13619.pem (1338 bytes)
I0216 17:28:19.730835 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/files/etc/ssl/certs/136192.pem --> /usr/share/ca-certificates/136192.pem (1708 bytes)
I0216 17:28:19.755808 354075 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-6821/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0216 17:28:19.782159 354075 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0216 17:28:19.799828 354075 ssh_runner.go:195] Run: openssl version
I0216 17:28:19.805439 354075 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/13619.pem && ln -fs /usr/share/ca-certificates/13619.pem /etc/ssl/certs/13619.pem"
I0216 17:28:19.815547 354075 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/13619.pem
I0216 17:28:19.819145 354075 certs.go:480] hashing: -rw-r--r-- 1 root root 1338 Feb 16 16:47 /usr/share/ca-certificates/13619.pem
I0216 17:28:19.819192 354075 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/13619.pem
I0216 17:28:19.825987 354075 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/13619.pem /etc/ssl/certs/51391683.0"
I0216 17:28:19.835186 354075 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/136192.pem && ln -fs /usr/share/ca-certificates/136192.pem /etc/ssl/certs/136192.pem"
I0216 17:28:19.844924 354075 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/136192.pem
I0216 17:28:19.848229 354075 certs.go:480] hashing: -rw-r--r-- 1 root root 1708 Feb 16 16:47 /usr/share/ca-certificates/136192.pem
I0216 17:28:19.848293 354075 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/136192.pem
I0216 17:28:19.854600 354075 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/136192.pem /etc/ssl/certs/3ec20f2e.0"
I0216 17:28:19.863466 354075 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0216 17:28:19.872672 354075 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0216 17:28:19.875826 354075 certs.go:480] hashing: -rw-r--r-- 1 root root 1111 Feb 16 16:43 /usr/share/ca-certificates/minikubeCA.pem
I0216 17:28:19.875879 354075 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0216 17:28:19.882063 354075 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0216 17:28:19.890858 354075 ssh_runner.go:195] Run: ls /var/lib/minikube/certs/etcd
I0216 17:28:19.894261 354075 certs.go:353] certs directory doesn't exist, likely first start: ls /var/lib/minikube/certs/etcd: Process exited with status 2
stdout:
stderr:
ls: cannot access '/var/lib/minikube/certs/etcd': No such file or directory
I0216 17:28:19.894334 354075 kubeadm.go:404] StartCluster: {Name:old-k8s-version-478853 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-478853 Namespace:default APIServerName:minikubeCA APISer
verNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:fals
e DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0216 17:28:19.894524 354075 ssh_runner.go:195] Run: docker ps --filter status=paused --filter=name=k8s_.*_(kube-system)_ --format={{.ID}}
I0216 17:28:19.911810 354075 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0216 17:28:19.920543 354075 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0216 17:28:19.929606 354075 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0216 17:28:19.929699 354075 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0216 17:28:19.938450 354075 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0216 17:28:19.938491 354075 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0216 17:28:19.993075 354075 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0216 17:28:19.993338 354075 kubeadm.go:322] [preflight] Running pre-flight checks
I0216 17:28:20.190335 354075 kubeadm.go:322] [preflight] The system verification failed. Printing the output from the verification:
I0216 17:28:20.190419 354075 kubeadm.go:322] [0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
I0216 17:28:20.190484 354075 kubeadm.go:322] [0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
I0216 17:28:20.190532 354075 kubeadm.go:322] [0;37mOS[0m: [0;32mLinux[0m
I0216 17:28:20.190593 354075 kubeadm.go:322] [0;37mCGROUPS_CPU[0m: [0;32menabled[0m
I0216 17:28:20.190661 354075 kubeadm.go:322] [0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
I0216 17:28:20.190732 354075 kubeadm.go:322] [0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
I0216 17:28:20.190810 354075 kubeadm.go:322] [0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
I0216 17:28:20.190878 354075 kubeadm.go:322] [0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
I0216 17:28:20.190945 354075 kubeadm.go:322] [0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
I0216 17:28:20.287337 354075 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0216 17:28:20.287474 354075 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0216 17:28:20.287638 354075 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0216 17:28:20.504570 354075 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0216 17:28:20.505180 354075 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0216 17:28:20.514204 354075 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0216 17:28:20.606927 354075 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0216 17:28:20.609215 354075 out.go:204] - Generating certificates and keys ...
I0216 17:28:20.609322 354075 kubeadm.go:322] [certs] Using existing ca certificate authority
I0216 17:28:20.610009 354075 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0216 17:28:20.887871 354075 kubeadm.go:322] [certs] Generating "apiserver-kubelet-client" certificate and key
I0216 17:28:20.989434 354075 kubeadm.go:322] [certs] Generating "front-proxy-ca" certificate and key
I0216 17:28:21.264754 354075 kubeadm.go:322] [certs] Generating "front-proxy-client" certificate and key
I0216 17:28:21.534612 354075 kubeadm.go:322] [certs] Generating "etcd/ca" certificate and key
I0216 17:28:21.784325 354075 kubeadm.go:322] [certs] Generating "etcd/server" certificate and key
I0216 17:28:21.784496 354075 kubeadm.go:322] [certs] etcd/server serving cert is signed for DNS names [old-k8s-version-478853 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
I0216 17:28:21.936895 354075 kubeadm.go:322] [certs] Generating "etcd/peer" certificate and key
I0216 17:28:21.937056 354075 kubeadm.go:322] [certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-478853 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
I0216 17:28:22.092848 354075 kubeadm.go:322] [certs] Generating "etcd/healthcheck-client" certificate and key
I0216 17:28:22.262825 354075 kubeadm.go:322] [certs] Generating "apiserver-etcd-client" certificate and key
I0216 17:28:22.384802 354075 kubeadm.go:322] [certs] Generating "sa" key and public key
I0216 17:28:22.385001 354075 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0216 17:28:22.450768 354075 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0216 17:28:22.641155 354075 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0216 17:28:22.717862 354075 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0216 17:28:22.828946 354075 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0216 17:28:22.829950 354075 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0216 17:28:22.831882 354075 out.go:204] - Booting up control plane ...
I0216 17:28:22.832014 354075 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0216 17:28:22.836593 354075 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0216 17:28:22.837646 354075 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0216 17:28:22.839641 354075 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0216 17:28:22.841892 354075 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0216 17:29:02.842106 354075 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0216 17:32:22.843255 354075 kubeadm.go:322]
I0216 17:32:22.843339 354075 kubeadm.go:322] Unfortunately, an error has occurred:
I0216 17:32:22.843400 354075 kubeadm.go:322] timed out waiting for the condition
I0216 17:32:22.843427 354075 kubeadm.go:322]
I0216 17:32:22.843480 354075 kubeadm.go:322] This error is likely caused by:
I0216 17:32:22.843514 354075 kubeadm.go:322] - The kubelet is not running
I0216 17:32:22.843649 354075 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0216 17:32:22.843674 354075 kubeadm.go:322]
I0216 17:32:22.843798 354075 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0216 17:32:22.843842 354075 kubeadm.go:322] - 'systemctl status kubelet'
I0216 17:32:22.843873 354075 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0216 17:32:22.843879 354075 kubeadm.go:322]
I0216 17:32:22.843966 354075 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0216 17:32:22.844089 354075 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0216 17:32:22.844249 354075 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0216 17:32:22.844311 354075 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0216 17:32:22.844394 354075 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0216 17:32:22.844425 354075 kubeadm.go:322] - 'docker logs CONTAINERID'
I0216 17:32:22.846928 354075 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0216 17:32:22.847077 354075 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
I0216 17:32:22.847279 354075 kubeadm.go:322] [WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
I0216 17:32:22.847435 354075 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0216 17:32:22.847514 354075 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0216 17:32:22.847570 354075 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
W0216 17:32:22.847728 354075 out.go:239] ! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-478853 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-478853 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-478853 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-478853 localhost] and IPs [192.168.76.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
I0216 17:32:22.847798 354075 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm reset --cri-socket /var/run/dockershim.sock --force"
I0216 17:32:23.619026 354075 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service kubelet
I0216 17:32:23.632000 354075 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0216 17:32:23.632066 354075 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0216 17:32:23.640732 354075 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0216 17:32:23.640799 354075 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0216 17:32:23.694213 354075 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0216 17:32:23.694278 354075 kubeadm.go:322] [preflight] Running pre-flight checks
I0216 17:32:23.876136 354075 kubeadm.go:322] [preflight] The system verification failed. Printing the output from the verification:
I0216 17:32:23.876293 354075 kubeadm.go:322] [0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
I0216 17:32:23.876375 354075 kubeadm.go:322] [0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
I0216 17:32:23.876422 354075 kubeadm.go:322] [0;37mOS[0m: [0;32mLinux[0m
I0216 17:32:23.876497 354075 kubeadm.go:322] [0;37mCGROUPS_CPU[0m: [0;32menabled[0m
I0216 17:32:23.876563 354075 kubeadm.go:322] [0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
I0216 17:32:23.876647 354075 kubeadm.go:322] [0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
I0216 17:32:23.876696 354075 kubeadm.go:322] [0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
I0216 17:32:23.876773 354075 kubeadm.go:322] [0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
I0216 17:32:23.876844 354075 kubeadm.go:322] [0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
I0216 17:32:23.953503 354075 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0216 17:32:23.953635 354075 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0216 17:32:23.953796 354075 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0216 17:32:24.136932 354075 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0216 17:32:24.139148 354075 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0216 17:32:24.147005 354075 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0216 17:32:24.225711 354075 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0216 17:32:24.227948 354075 out.go:204] - Generating certificates and keys ...
I0216 17:32:24.228117 354075 kubeadm.go:322] [certs] Using existing ca certificate authority
I0216 17:32:24.228230 354075 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0216 17:32:24.228332 354075 kubeadm.go:322] [certs] Using existing apiserver-kubelet-client certificate and key on disk
I0216 17:32:24.228436 354075 kubeadm.go:322] [certs] Using existing front-proxy-ca certificate authority
I0216 17:32:24.228547 354075 kubeadm.go:322] [certs] Using existing front-proxy-client certificate and key on disk
I0216 17:32:24.228616 354075 kubeadm.go:322] [certs] Using existing etcd/ca certificate authority
I0216 17:32:24.228699 354075 kubeadm.go:322] [certs] Using existing etcd/server certificate and key on disk
I0216 17:32:24.228782 354075 kubeadm.go:322] [certs] Using existing etcd/peer certificate and key on disk
I0216 17:32:24.228908 354075 kubeadm.go:322] [certs] Using existing etcd/healthcheck-client certificate and key on disk
I0216 17:32:24.229013 354075 kubeadm.go:322] [certs] Using existing apiserver-etcd-client certificate and key on disk
I0216 17:32:24.229066 354075 kubeadm.go:322] [certs] Using the existing "sa" key
I0216 17:32:24.229137 354075 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0216 17:32:24.350573 354075 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0216 17:32:24.539801 354075 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0216 17:32:24.596237 354075 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0216 17:32:24.732946 354075 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0216 17:32:24.733959 354075 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0216 17:32:24.735941 354075 out.go:204] - Booting up control plane ...
I0216 17:32:24.736069 354075 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0216 17:32:24.742690 354075 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0216 17:32:24.744111 354075 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0216 17:32:24.745089 354075 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0216 17:32:24.747884 354075 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0216 17:33:04.748145 354075 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0216 17:36:24.749312 354075 kubeadm.go:322]
I0216 17:36:24.749414 354075 kubeadm.go:322] Unfortunately, an error has occurred:
I0216 17:36:24.749466 354075 kubeadm.go:322] timed out waiting for the condition
I0216 17:36:24.749472 354075 kubeadm.go:322]
I0216 17:36:24.749508 354075 kubeadm.go:322] This error is likely caused by:
I0216 17:36:24.749540 354075 kubeadm.go:322] - The kubelet is not running
I0216 17:36:24.749654 354075 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0216 17:36:24.749662 354075 kubeadm.go:322]
I0216 17:36:24.749813 354075 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0216 17:36:24.749883 354075 kubeadm.go:322] - 'systemctl status kubelet'
I0216 17:36:24.749920 354075 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0216 17:36:24.749930 354075 kubeadm.go:322]
I0216 17:36:24.750073 354075 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0216 17:36:24.750199 354075 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0216 17:36:24.750291 354075 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0216 17:36:24.750374 354075 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0216 17:36:24.750475 354075 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0216 17:36:24.750512 354075 kubeadm.go:322] - 'docker logs CONTAINERID'
I0216 17:36:24.753234 354075 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0216 17:36:24.753398 354075 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
I0216 17:36:24.753670 354075 kubeadm.go:322] [WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
I0216 17:36:24.753780 354075 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0216 17:36:24.753848 354075 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0216 17:36:24.753910 354075 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
I0216 17:36:24.754138 354075 kubeadm.go:406] StartCluster complete in 8m4.859809472s
I0216 17:36:24.754281 354075 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-apiserver --format={{.ID}}
I0216 17:36:24.775940 354075 logs.go:276] 0 containers: []
W0216 17:36:24.775979 354075 logs.go:278] No container was found matching "kube-apiserver"
I0216 17:36:24.776060 354075 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_etcd --format={{.ID}}
I0216 17:36:24.796468 354075 logs.go:276] 0 containers: []
W0216 17:36:24.796493 354075 logs.go:278] No container was found matching "etcd"
I0216 17:36:24.796558 354075 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_coredns --format={{.ID}}
I0216 17:36:24.815993 354075 logs.go:276] 0 containers: []
W0216 17:36:24.816014 354075 logs.go:278] No container was found matching "coredns"
I0216 17:36:24.816071 354075 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-scheduler --format={{.ID}}
I0216 17:36:24.835724 354075 logs.go:276] 0 containers: []
W0216 17:36:24.835753 354075 logs.go:278] No container was found matching "kube-scheduler"
I0216 17:36:24.835827 354075 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-proxy --format={{.ID}}
I0216 17:36:24.855999 354075 logs.go:276] 0 containers: []
W0216 17:36:24.856023 354075 logs.go:278] No container was found matching "kube-proxy"
I0216 17:36:24.856078 354075 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-controller-manager --format={{.ID}}
I0216 17:36:24.877784 354075 logs.go:276] 0 containers: []
W0216 17:36:24.877812 354075 logs.go:278] No container was found matching "kube-controller-manager"
I0216 17:36:24.877865 354075 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kindnet --format={{.ID}}
I0216 17:36:24.895310 354075 logs.go:276] 0 containers: []
W0216 17:36:24.895330 354075 logs.go:278] No container was found matching "kindnet"
I0216 17:36:24.895340 354075 logs.go:123] Gathering logs for kubelet ...
I0216 17:36:24.895354 354075 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0216 17:36:24.921097 354075 logs.go:138] Found kubelet problem: Feb 16 17:36:03 old-k8s-version-478853 kubelet[5716]: E0216 17:36:03.828321 5716 pod_workers.go:191] Error syncing pod 75eac6fd65f4f8477f5572974a7da828 ("etcd-old-k8s-version-478853_kube-system(75eac6fd65f4f8477f5572974a7da828)"), skipping: failed to "StartContainer" for "etcd" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/etcd:3.3.15-0\": Id or size of image \"k8s.gcr.io/etcd:3.3.15-0\" is not set"
W0216 17:36:24.923305 354075 logs.go:138] Found kubelet problem: Feb 16 17:36:04 old-k8s-version-478853 kubelet[5716]: E0216 17:36:04.833813 5716 pod_workers.go:191] Error syncing pod 9fc427d2e6746d2b3f18846f6f0fcafb ("kube-controller-manager-old-k8s-version-478853_kube-system(9fc427d2e6746d2b3f18846f6f0fcafb)"), skipping: failed to "StartContainer" for "kube-controller-manager" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-controller-manager:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-controller-manager:v1.16.0\" is not set"
W0216 17:36:24.927005 354075 logs.go:138] Found kubelet problem: Feb 16 17:36:06 old-k8s-version-478853 kubelet[5716]: E0216 17:36:06.833227 5716 pod_workers.go:191] Error syncing pod b3d303074fe0ca1d42a8bd9ed248df09 ("kube-scheduler-old-k8s-version-478853_kube-system(b3d303074fe0ca1d42a8bd9ed248df09)"), skipping: failed to "StartContainer" for "kube-scheduler" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-scheduler:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-scheduler:v1.16.0\" is not set"
W0216 17:36:24.929415 354075 logs.go:138] Found kubelet problem: Feb 16 17:36:07 old-k8s-version-478853 kubelet[5716]: E0216 17:36:07.822892 5716 pod_workers.go:191] Error syncing pod 002009a6866b0a2506f8d5c8c4da7548 ("kube-apiserver-old-k8s-version-478853_kube-system(002009a6866b0a2506f8d5c8c4da7548)"), skipping: failed to "StartContainer" for "kube-apiserver" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-apiserver:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-apiserver:v1.16.0\" is not set"
W0216 17:36:24.940745 354075 logs.go:138] Found kubelet problem: Feb 16 17:36:14 old-k8s-version-478853 kubelet[5716]: E0216 17:36:14.821265 5716 pod_workers.go:191] Error syncing pod 75eac6fd65f4f8477f5572974a7da828 ("etcd-old-k8s-version-478853_kube-system(75eac6fd65f4f8477f5572974a7da828)"), skipping: failed to "StartContainer" for "etcd" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/etcd:3.3.15-0\": Id or size of image \"k8s.gcr.io/etcd:3.3.15-0\" is not set"
W0216 17:36:24.946152 354075 logs.go:138] Found kubelet problem: Feb 16 17:36:17 old-k8s-version-478853 kubelet[5716]: E0216 17:36:17.828884 5716 pod_workers.go:191] Error syncing pod b3d303074fe0ca1d42a8bd9ed248df09 ("kube-scheduler-old-k8s-version-478853_kube-system(b3d303074fe0ca1d42a8bd9ed248df09)"), skipping: failed to "StartContainer" for "kube-scheduler" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-scheduler:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-scheduler:v1.16.0\" is not set"
W0216 17:36:24.948239 354075 logs.go:138] Found kubelet problem: Feb 16 17:36:18 old-k8s-version-478853 kubelet[5716]: E0216 17:36:18.845816 5716 pod_workers.go:191] Error syncing pod 002009a6866b0a2506f8d5c8c4da7548 ("kube-apiserver-old-k8s-version-478853_kube-system(002009a6866b0a2506f8d5c8c4da7548)"), skipping: failed to "StartContainer" for "kube-apiserver" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-apiserver:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-apiserver:v1.16.0\" is not set"
W0216 17:36:24.948726 354075 logs.go:138] Found kubelet problem: Feb 16 17:36:18 old-k8s-version-478853 kubelet[5716]: E0216 17:36:18.848783 5716 pod_workers.go:191] Error syncing pod 9fc427d2e6746d2b3f18846f6f0fcafb ("kube-controller-manager-old-k8s-version-478853_kube-system(9fc427d2e6746d2b3f18846f6f0fcafb)"), skipping: failed to "StartContainer" for "kube-controller-manager" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-controller-manager:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-controller-manager:v1.16.0\" is not set"
I0216 17:36:24.957722 354075 logs.go:123] Gathering logs for dmesg ...
I0216 17:36:24.957748 354075 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0216 17:36:24.983071 354075 logs.go:123] Gathering logs for describe nodes ...
I0216 17:36:24.983168 354075 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0216 17:36:25.062664 354075 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0216 17:36:25.062689 354075 logs.go:123] Gathering logs for Docker ...
I0216 17:36:25.062699 354075 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u docker -u cri-docker -n 400"
I0216 17:36:25.082257 354075 logs.go:123] Gathering logs for container status ...
I0216 17:36:25.082296 354075 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
W0216 17:36:25.120328 354075 out.go:364] Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0216 17:36:25.120385 354075 out.go:239] *
*
W0216 17:36:25.120450 354075 out.go:239] X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0216 17:36:25.120485 354075 out.go:239] *
*
W0216 17:36:25.121556 354075 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0216 17:36:25.123694 354075 out.go:177] X Problems detected in kubelet:
I0216 17:36:25.125393 354075 out.go:177] Feb 16 17:36:03 old-k8s-version-478853 kubelet[5716]: E0216 17:36:03.828321 5716 pod_workers.go:191] Error syncing pod 75eac6fd65f4f8477f5572974a7da828 ("etcd-old-k8s-version-478853_kube-system(75eac6fd65f4f8477f5572974a7da828)"), skipping: failed to "StartContainer" for "etcd" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/etcd:3.3.15-0\": Id or size of image \"k8s.gcr.io/etcd:3.3.15-0\" is not set"
I0216 17:36:25.126998 354075 out.go:177] Feb 16 17:36:04 old-k8s-version-478853 kubelet[5716]: E0216 17:36:04.833813 5716 pod_workers.go:191] Error syncing pod 9fc427d2e6746d2b3f18846f6f0fcafb ("kube-controller-manager-old-k8s-version-478853_kube-system(9fc427d2e6746d2b3f18846f6f0fcafb)"), skipping: failed to "StartContainer" for "kube-controller-manager" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-controller-manager:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-controller-manager:v1.16.0\" is not set"
I0216 17:36:25.128619 354075 out.go:177] Feb 16 17:36:06 old-k8s-version-478853 kubelet[5716]: E0216 17:36:06.833227 5716 pod_workers.go:191] Error syncing pod b3d303074fe0ca1d42a8bd9ed248df09 ("kube-scheduler-old-k8s-version-478853_kube-system(b3d303074fe0ca1d42a8bd9ed248df09)"), skipping: failed to "StartContainer" for "kube-scheduler" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-scheduler:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-scheduler:v1.16.0\" is not set"
I0216 17:36:25.131383 354075 out.go:177]
W0216 17:36:25.132773 354075 out.go:239] X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1051-gcp[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1051-gcp\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0216 17:36:25.132828 354075 out.go:239] * Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
* Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
W0216 17:36:25.132849 354075 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/4172
* Related issue: https://github.com/kubernetes/minikube/issues/4172
I0216 17:36:25.134560 354075 out.go:177]
** /stderr **
start_stop_delete_test.go:188: failed starting minikube -first start-. args "out/minikube-linux-amd64 start -p old-k8s-version-478853 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --container-runtime=docker --kubernetes-version=v1.16.0": exit status 109
helpers_test.go:222: -----------------------post-mortem--------------------------------
helpers_test.go:230: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/FirstStart]: docker inspect <======
helpers_test.go:231: (dbg) Run: docker inspect old-k8s-version-478853
helpers_test.go:235: (dbg) docker inspect old-k8s-version-478853:
-- stdout --
[
{
"Id": "74b66ed59b2b04b54f2d2a9f2d5252a296723d9f3a251b88b2bb07496976cfde",
"Created": "2024-02-16T17:28:05.344964673Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 355623,
"ExitCode": 0,
"Error": "",
"StartedAt": "2024-02-16T17:28:05.697815646Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:78bbddd92c7656e5a7bf9651f59e6b47aa97241efd2e93247c457ec76b2185c5",
"ResolvConfPath": "/var/lib/docker/containers/74b66ed59b2b04b54f2d2a9f2d5252a296723d9f3a251b88b2bb07496976cfde/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/74b66ed59b2b04b54f2d2a9f2d5252a296723d9f3a251b88b2bb07496976cfde/hostname",
"HostsPath": "/var/lib/docker/containers/74b66ed59b2b04b54f2d2a9f2d5252a296723d9f3a251b88b2bb07496976cfde/hosts",
"LogPath": "/var/lib/docker/containers/74b66ed59b2b04b54f2d2a9f2d5252a296723d9f3a251b88b2bb07496976cfde/74b66ed59b2b04b54f2d2a9f2d5252a296723d9f3a251b88b2bb07496976cfde-json.log",
"Name": "/old-k8s-version-478853",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "unconfined",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro",
"old-k8s-version-478853:/var"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {
"max-size": "100m"
}
},
"NetworkMode": "old-k8s-version-478853",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "host",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 2306867200,
"NanoCpus": 2000000000,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": [],
"BlkioDeviceWriteBps": [],
"BlkioDeviceReadIOps": [],
"BlkioDeviceWriteIOps": [],
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 4613734400,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": [],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/e8b746ed1c7e2739d6ff7faf8fc718f1484cabd23b642a71f94e72690435a083-init/diff:/var/lib/docker/overlay2/399457765d8a71bf3b9151eb69e824afe917f6f0e4f38614a9c00a72b38b806a/diff",
"MergedDir": "/var/lib/docker/overlay2/e8b746ed1c7e2739d6ff7faf8fc718f1484cabd23b642a71f94e72690435a083/merged",
"UpperDir": "/var/lib/docker/overlay2/e8b746ed1c7e2739d6ff7faf8fc718f1484cabd23b642a71f94e72690435a083/diff",
"WorkDir": "/var/lib/docker/overlay2/e8b746ed1c7e2739d6ff7faf8fc718f1484cabd23b642a71f94e72690435a083/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "old-k8s-version-478853",
"Source": "/var/lib/docker/volumes/old-k8s-version-478853/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "old-k8s-version-478853",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"22/tcp": {},
"2376/tcp": {},
"32443/tcp": {},
"5000/tcp": {},
"8443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "old-k8s-version-478853",
"name.minikube.sigs.k8s.io": "old-k8s-version-478853",
"role.minikube.sigs.k8s.io": ""
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "2f50c95616da1165e1cfabfc149c9e4fe9ac50a2d326751a0e496eb203737e29",
"SandboxKey": "/var/run/docker/netns/2f50c95616da",
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33052"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33051"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33048"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33050"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33049"
}
]
},
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"old-k8s-version-478853": {
"IPAMConfig": {
"IPv4Address": "192.168.76.2"
},
"Links": null,
"Aliases": [
"74b66ed59b2b",
"old-k8s-version-478853"
],
"MacAddress": "02:42:c0:a8:4c:02",
"NetworkID": "166a9b0cbcbad81945e5ddf7b3ae3a6fed94ef48dba3d7d6ceb648c91593d0fb",
"EndpointID": "ed2f7497eca108063ac7a9de2d606faa1c6b0a335757a0d57db71815983a1865",
"Gateway": "192.168.76.1",
"IPAddress": "192.168.76.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"DriverOpts": null,
"DNSNames": [
"old-k8s-version-478853",
"74b66ed59b2b"
]
}
}
}
}
]
-- /stdout --
helpers_test.go:239: (dbg) Run: out/minikube-linux-amd64 status --format={{.Host}} -p old-k8s-version-478853 -n old-k8s-version-478853
helpers_test.go:239: (dbg) Non-zero exit: out/minikube-linux-amd64 status --format={{.Host}} -p old-k8s-version-478853 -n old-k8s-version-478853: exit status 6 (328.490823ms)
-- stdout --
Running
WARNING: Your kubectl is pointing to stale minikube-vm.
To fix the kubectl context, run `minikube update-context`
-- /stdout --
** stderr **
E0216 17:36:25.520734 438552 status.go:415] kubeconfig endpoint: extract IP: "old-k8s-version-478853" does not appear in /home/jenkins/minikube-integration/17936-6821/kubeconfig
** /stderr **
helpers_test.go:239: status error: exit status 6 (may be ok)
helpers_test.go:241: "old-k8s-version-478853" host is not running, skipping log retrieval (state="Running\nWARNING: Your kubectl is pointing to stale minikube-vm.\nTo fix the kubectl context, run `minikube update-context`")
--- FAIL: TestStartStop/group/old-k8s-version/serial/FirstStart (504.60s)