=== RUN TestStartStop/group/old-k8s-version/serial/FirstStart
start_stop_delete_test.go:186: (dbg) Run: out/minikube-linux-arm64 start -p old-k8s-version-488384 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --container-runtime=docker --kubernetes-version=v1.16.0
E0216 17:43:47.461672 7513 cert_rotation.go:168] key failed with : open /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/calico-850655/client.crt: no such file or directory
E0216 17:43:57.702116 7513 cert_rotation.go:168] key failed with : open /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/calico-850655/client.crt: no such file or directory
E0216 17:44:02.148794 7513 cert_rotation.go:168] key failed with : open /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/kindnet-850655/client.crt: no such file or directory
E0216 17:44:15.158307 7513 cert_rotation.go:168] key failed with : open /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/auto-850655/client.crt: no such file or directory
E0216 17:44:18.182956 7513 cert_rotation.go:168] key failed with : open /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/calico-850655/client.crt: no such file or directory
E0216 17:44:43.109166 7513 cert_rotation.go:168] key failed with : open /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/kindnet-850655/client.crt: no such file or directory
E0216 17:44:50.355172 7513 cert_rotation.go:168] key failed with : open /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/addons-105162/client.crt: no such file or directory
start_stop_delete_test.go:186: (dbg) Non-zero exit: out/minikube-linux-arm64 start -p old-k8s-version-488384 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --container-runtime=docker --kubernetes-version=v1.16.0: exit status 109 (8m36.557138684s)
-- stdout --
* [old-k8s-version-488384] minikube v1.32.0 on Ubuntu 20.04 (arm64)
- MINIKUBE_LOCATION=17936
- MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
- KUBECONFIG=/home/jenkins/minikube-integration/17936-2208/kubeconfig
- MINIKUBE_HOME=/home/jenkins/minikube-integration/17936-2208/.minikube
- MINIKUBE_BIN=out/minikube-linux-arm64
- MINIKUBE_FORCE_SYSTEMD=
* Using the docker driver based on user configuration
* Using Docker driver with root privileges
* Starting control plane node old-k8s-version-488384 in cluster old-k8s-version-488384
* Pulling base image v0.0.42-1708008208-17936 ...
* Creating docker container (CPUs=2, Memory=2200MB) ...
* Preparing Kubernetes v1.16.0 on Docker 25.0.3 ...
- Generating certificates and keys ...
- Booting up control plane ...
- Generating certificates and keys ...
- Booting up control plane ...
X Problems detected in kubelet:
Feb 16 17:52:01 old-k8s-version-488384 kubelet[5456]: E0216 17:52:01.519546 5456 pod_workers.go:191] Error syncing pod 9e914279041b7fec183513988f6a94cb ("kube-apiserver-old-k8s-version-488384_kube-system(9e914279041b7fec183513988f6a94cb)"), skipping: failed to "StartContainer" for "kube-apiserver" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-apiserver:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-apiserver:v1.16.0\" is not set"
Feb 16 17:52:06 old-k8s-version-488384 kubelet[5456]: E0216 17:52:06.513912 5456 pod_workers.go:191] Error syncing pod f5bdbfcedf2bccc429ba471ffe3804b7 ("etcd-old-k8s-version-488384_kube-system(f5bdbfcedf2bccc429ba471ffe3804b7)"), skipping: failed to "StartContainer" for "etcd" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/etcd:3.3.15-0\": Id or size of image \"k8s.gcr.io/etcd:3.3.15-0\" is not set"
Feb 16 17:52:07 old-k8s-version-488384 kubelet[5456]: E0216 17:52:07.514080 5456 pod_workers.go:191] Error syncing pod b3d303074fe0ca1d42a8bd9ed248df09 ("kube-scheduler-old-k8s-version-488384_kube-system(b3d303074fe0ca1d42a8bd9ed248df09)"), skipping: failed to "StartContainer" for "kube-scheduler" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-scheduler:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-scheduler:v1.16.0\" is not set"
-- /stdout --
** stderr **
I0216 17:43:45.917338 305464 out.go:291] Setting OutFile to fd 1 ...
I0216 17:43:45.917600 305464 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0216 17:43:45.917626 305464 out.go:304] Setting ErrFile to fd 2...
I0216 17:43:45.917646 305464 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0216 17:43:45.917936 305464 root.go:338] Updating PATH: /home/jenkins/minikube-integration/17936-2208/.minikube/bin
I0216 17:43:45.918419 305464 out.go:298] Setting JSON to false
I0216 17:43:45.919431 305464 start.go:129] hostinfo: {"hostname":"ip-172-31-30-239","uptime":5176,"bootTime":1708100250,"procs":205,"os":"linux","platform":"ubuntu","platformFamily":"debian","platformVersion":"20.04","kernelVersion":"5.15.0-1053-aws","kernelArch":"aarch64","virtualizationSystem":"","virtualizationRole":"","hostId":"92f46a7d-c249-4c12-924a-77f64874c910"}
I0216 17:43:45.919522 305464 start.go:139] virtualization:
I0216 17:43:45.922579 305464 out.go:177] * [old-k8s-version-488384] minikube v1.32.0 on Ubuntu 20.04 (arm64)
I0216 17:43:45.924790 305464 out.go:177] - MINIKUBE_LOCATION=17936
I0216 17:43:45.926775 305464 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0216 17:43:45.924840 305464 notify.go:220] Checking for updates...
I0216 17:43:45.928743 305464 out.go:177] - KUBECONFIG=/home/jenkins/minikube-integration/17936-2208/kubeconfig
I0216 17:43:45.930916 305464 out.go:177] - MINIKUBE_HOME=/home/jenkins/minikube-integration/17936-2208/.minikube
I0216 17:43:45.932780 305464 out.go:177] - MINIKUBE_BIN=out/minikube-linux-arm64
I0216 17:43:45.934565 305464 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0216 17:43:45.936945 305464 config.go:182] Loaded profile config "kubenet-850655": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.28.4
I0216 17:43:45.937049 305464 driver.go:392] Setting default libvirt URI to qemu:///system
I0216 17:43:45.962100 305464 docker.go:122] docker version: linux-25.0.3:Docker Engine - Community
I0216 17:43:45.962211 305464 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0216 17:43:46.089994 305464 info.go:266] docker info: {ID:6ZPO:QZND:VNGE:LUKL:4Y3K:XELL:AAX4:2GTK:E6LM:MPRN:3ZXR:TTMR Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:32 OomKillDisable:true NGoroutines:56 SystemTime:2024-02-16 17:43:46.077191951 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1053-aws OperatingSystem:Ubuntu 20.04.6 LTS OSType:linux Architecture:aar
ch64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:2 MemTotal:8215044096 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ip-172-31-30-239 Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=builtin] ProductLicense: Warnings:<nil> ServerErro
rs:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.5]] Warnings:<nil>}}
I0216 17:43:46.090094 305464 docker.go:295] overlay module found
I0216 17:43:46.092699 305464 out.go:177] * Using the docker driver based on user configuration
I0216 17:43:46.094830 305464 start.go:299] selected driver: docker
I0216 17:43:46.094844 305464 start.go:903] validating driver "docker" against <nil>
I0216 17:43:46.094862 305464 start.go:914] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0216 17:43:46.095465 305464 cli_runner.go:164] Run: docker system info --format "{{json .}}"
I0216 17:43:46.185702 305464 info.go:266] docker info: {ID:6ZPO:QZND:VNGE:LUKL:4Y3K:XELL:AAX4:2GTK:E6LM:MPRN:3ZXR:TTMR Containers:1 ContainersRunning:1 ContainersPaused:0 ContainersStopped:0 Images:4 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Using metacopy false] [Native Overlay Diff true] [userxattr false]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:false KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:32 OomKillDisable:true NGoroutines:56 SystemTime:2024-02-16 17:43:46.175132129 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.15.0-1053-aws OperatingSystem:Ubuntu 20.04.6 LTS OSType:linux Architecture:aar
ch64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:2 MemTotal:8215044096 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:ip-172-31-30-239 Labels:[] ExperimentalBuild:false ServerVersion:25.0.3 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:ae07eda36dd25f8a1b98dfbf587313b99c0190bb Expected:ae07eda36dd25f8a1b98dfbf587313b99c0190bb} RuncCommit:{ID:v1.1.12-0-g51d5e94 Expected:v1.1.12-0-g51d5e94} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=apparmor name=seccomp,profile=builtin] ProductLicense: Warnings:<nil> ServerErro
rs:[] ClientInfo:{Debug:false Plugins:[map[Name:buildx Path:/usr/libexec/docker/cli-plugins/docker-buildx SchemaVersion:0.1.0 ShortDescription:Docker Buildx Vendor:Docker Inc. Version:v0.12.1] map[Name:compose Path:/usr/libexec/docker/cli-plugins/docker-compose SchemaVersion:0.1.0 ShortDescription:Docker Compose Vendor:Docker Inc. Version:v2.24.5]] Warnings:<nil>}}
I0216 17:43:46.185852 305464 start_flags.go:309] no existing cluster config was found, will generate one from the flags
I0216 17:43:46.186066 305464 start_flags.go:931] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0216 17:43:46.188033 305464 out.go:177] * Using Docker driver with root privileges
I0216 17:43:46.189546 305464 cni.go:84] Creating CNI manager for ""
I0216 17:43:46.189568 305464 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0216 17:43:46.189579 305464 start_flags.go:323] config:
{Name:old-k8s-version-488384 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-488384 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local Containe
rRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0216 17:43:46.193000 305464 out.go:177] * Starting control plane node old-k8s-version-488384 in cluster old-k8s-version-488384
I0216 17:43:46.194846 305464 cache.go:121] Beginning downloading kic base image for docker with docker
I0216 17:43:46.196751 305464 out.go:177] * Pulling base image v0.0.42-1708008208-17936 ...
I0216 17:43:46.198455 305464 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0216 17:43:46.198499 305464 preload.go:148] Found local preload: /home/jenkins/minikube-integration/17936-2208/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-arm64.tar.lz4
I0216 17:43:46.198513 305464 cache.go:56] Caching tarball of preloaded images
I0216 17:43:46.198587 305464 preload.go:174] Found /home/jenkins/minikube-integration/17936-2208/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-arm64.tar.lz4 in cache, skipping download
I0216 17:43:46.198597 305464 cache.go:59] Finished verifying existence of preloaded tar for v1.16.0 on docker
I0216 17:43:46.198699 305464 profile.go:148] Saving config to /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/config.json ...
I0216 17:43:46.198717 305464 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/config.json: {Name:mk67edff72f078cc3d0b50c1c7c7c02b0c181363 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:43:46.198857 305464 image.go:79] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf in local docker daemon
I0216 17:43:46.228038 305464 image.go:83] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf in local docker daemon, skipping pull
I0216 17:43:46.228058 305464 cache.go:144] gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf exists in daemon, skipping load
I0216 17:43:46.228075 305464 cache.go:194] Successfully downloaded all kic artifacts
I0216 17:43:46.228102 305464 start.go:365] acquiring machines lock for old-k8s-version-488384: {Name:mk24b3849756b4e9198a885ebb33ae78e58e4fe1 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0216 17:43:46.228212 305464 start.go:369] acquired machines lock for "old-k8s-version-488384" in 84.382µs
I0216 17:43:46.228238 305464 start.go:93] Provisioning new machine with config: &{Name:old-k8s-version-488384 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-488384 Namespace:default APIServer
Name:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimiza
tions:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s} &{Name: IP: Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}
I0216 17:43:46.228311 305464 start.go:125] createHost starting for "" (driver="docker")
I0216 17:43:46.230423 305464 out.go:204] * Creating docker container (CPUs=2, Memory=2200MB) ...
I0216 17:43:46.230665 305464 start.go:159] libmachine.API.Create for "old-k8s-version-488384" (driver="docker")
I0216 17:43:46.230697 305464 client.go:168] LocalClient.Create starting
I0216 17:43:46.230783 305464 main.go:141] libmachine: Reading certificate data from /home/jenkins/minikube-integration/17936-2208/.minikube/certs/ca.pem
I0216 17:43:46.230814 305464 main.go:141] libmachine: Decoding PEM data...
I0216 17:43:46.230828 305464 main.go:141] libmachine: Parsing certificate...
I0216 17:43:46.230875 305464 main.go:141] libmachine: Reading certificate data from /home/jenkins/minikube-integration/17936-2208/.minikube/certs/cert.pem
I0216 17:43:46.230891 305464 main.go:141] libmachine: Decoding PEM data...
I0216 17:43:46.230905 305464 main.go:141] libmachine: Parsing certificate...
I0216 17:43:46.231264 305464 cli_runner.go:164] Run: docker network inspect old-k8s-version-488384 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
W0216 17:43:46.256101 305464 cli_runner.go:211] docker network inspect old-k8s-version-488384 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1
I0216 17:43:46.256175 305464 network_create.go:281] running [docker network inspect old-k8s-version-488384] to gather additional debugging logs...
I0216 17:43:46.256201 305464 cli_runner.go:164] Run: docker network inspect old-k8s-version-488384
W0216 17:43:46.275288 305464 cli_runner.go:211] docker network inspect old-k8s-version-488384 returned with exit code 1
I0216 17:43:46.275313 305464 network_create.go:284] error running [docker network inspect old-k8s-version-488384]: docker network inspect old-k8s-version-488384: exit status 1
stdout:
[]
stderr:
Error response from daemon: network old-k8s-version-488384 not found
I0216 17:43:46.275326 305464 network_create.go:286] output of [docker network inspect old-k8s-version-488384]: -- stdout --
[]
-- /stdout --
** stderr **
Error response from daemon: network old-k8s-version-488384 not found
** /stderr **
I0216 17:43:46.275420 305464 cli_runner.go:164] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0216 17:43:46.299165 305464 network.go:212] skipping subnet 192.168.49.0/24 that is taken: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName:br-bf2219ceb1d4 IfaceIPv4:192.168.49.1 IfaceMTU:1500 IfaceMAC:02:42:fc:0a:69:d6} reservation:<nil>}
I0216 17:43:46.299565 305464 network.go:212] skipping subnet 192.168.58.0/24 that is taken: &{IP:192.168.58.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.58.0/24 Gateway:192.168.58.1 ClientMin:192.168.58.2 ClientMax:192.168.58.254 Broadcast:192.168.58.255 IsPrivate:true Interface:{IfaceName:br-88cc490de1c4 IfaceIPv4:192.168.58.1 IfaceMTU:1500 IfaceMAC:02:42:6d:6b:26:04} reservation:<nil>}
I0216 17:43:46.300022 305464 network.go:207] using free private subnet 192.168.67.0/24: &{IP:192.168.67.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.67.0/24 Gateway:192.168.67.1 ClientMin:192.168.67.2 ClientMax:192.168.67.254 Broadcast:192.168.67.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0x40024fbce0}
I0216 17:43:46.300041 305464 network_create.go:124] attempt to create docker network old-k8s-version-488384 192.168.67.0/24 with gateway 192.168.67.1 and MTU of 1500 ...
I0216 17:43:46.300093 305464 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.67.0/24 --gateway=192.168.67.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=1500 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=old-k8s-version-488384 old-k8s-version-488384
I0216 17:43:46.381645 305464 network_create.go:108] docker network old-k8s-version-488384 192.168.67.0/24 created
I0216 17:43:46.381672 305464 kic.go:121] calculated static IP "192.168.67.2" for the "old-k8s-version-488384" container
I0216 17:43:46.381753 305464 cli_runner.go:164] Run: docker ps -a --format {{.Names}}
I0216 17:43:46.402695 305464 cli_runner.go:164] Run: docker volume create old-k8s-version-488384 --label name.minikube.sigs.k8s.io=old-k8s-version-488384 --label created_by.minikube.sigs.k8s.io=true
I0216 17:43:46.419911 305464 oci.go:103] Successfully created a docker volume old-k8s-version-488384
I0216 17:43:46.420002 305464 cli_runner.go:164] Run: docker run --rm --name old-k8s-version-488384-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-488384 --entrypoint /usr/bin/test -v old-k8s-version-488384:/var gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf -d /var/lib
I0216 17:43:47.103904 305464 oci.go:107] Successfully prepared a docker volume old-k8s-version-488384
I0216 17:43:47.103958 305464 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0216 17:43:47.103979 305464 kic.go:194] Starting extracting preloaded images to volume ...
I0216 17:43:47.104057 305464 cli_runner.go:164] Run: docker run --rm --entrypoint /usr/bin/tar -v /home/jenkins/minikube-integration/17936-2208/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-arm64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-488384:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf -I lz4 -xf /preloaded.tar -C /extractDir
I0216 17:43:51.651289 305464 cli_runner.go:217] Completed: docker run --rm --entrypoint /usr/bin/tar -v /home/jenkins/minikube-integration/17936-2208/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-arm64.tar.lz4:/preloaded.tar:ro -v old-k8s-version-488384:/extractDir gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf -I lz4 -xf /preloaded.tar -C /extractDir: (4.547193135s)
I0216 17:43:51.651316 305464 kic.go:203] duration metric: took 4.547335 seconds to extract preloaded images to volume
W0216 17:43:51.651452 305464 cgroups_linux.go:77] Your kernel does not support swap limit capabilities or the cgroup is not mounted.
I0216 17:43:51.651554 305464 cli_runner.go:164] Run: docker info --format "'{{json .SecurityOptions}}'"
I0216 17:43:51.763585 305464 cli_runner.go:164] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname old-k8s-version-488384 --name old-k8s-version-488384 --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=old-k8s-version-488384 --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=old-k8s-version-488384 --network old-k8s-version-488384 --ip 192.168.67.2 --volume old-k8s-version-488384:/var --security-opt apparmor=unconfined --memory=2200mb --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf
I0216 17:43:52.330643 305464 cli_runner.go:164] Run: docker container inspect old-k8s-version-488384 --format={{.State.Running}}
I0216 17:43:52.357379 305464 cli_runner.go:164] Run: docker container inspect old-k8s-version-488384 --format={{.State.Status}}
I0216 17:43:52.390564 305464 cli_runner.go:164] Run: docker exec old-k8s-version-488384 stat /var/lib/dpkg/alternatives/iptables
I0216 17:43:52.459439 305464 oci.go:144] the created container "old-k8s-version-488384" has a running status.
I0216 17:43:52.459468 305464 kic.go:225] Creating ssh key for kic: /home/jenkins/minikube-integration/17936-2208/.minikube/machines/old-k8s-version-488384/id_rsa...
I0216 17:43:53.045839 305464 kic_runner.go:191] docker (temp): /home/jenkins/minikube-integration/17936-2208/.minikube/machines/old-k8s-version-488384/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0216 17:43:53.071578 305464 cli_runner.go:164] Run: docker container inspect old-k8s-version-488384 --format={{.State.Status}}
I0216 17:43:53.094255 305464 kic_runner.go:93] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0216 17:43:53.094277 305464 kic_runner.go:114] Args: [docker exec --privileged old-k8s-version-488384 chown docker:docker /home/docker/.ssh/authorized_keys]
I0216 17:43:53.186113 305464 cli_runner.go:164] Run: docker container inspect old-k8s-version-488384 --format={{.State.Status}}
I0216 17:43:53.228820 305464 machine.go:88] provisioning docker machine ...
I0216 17:43:53.228862 305464 ubuntu.go:169] provisioning hostname "old-k8s-version-488384"
I0216 17:43:53.228930 305464 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-488384
I0216 17:43:53.252315 305464 main.go:141] libmachine: Using SSH client type: native
I0216 17:43:53.252892 305464 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3bf490] 0x3c1c00 <nil> [] 0s} 127.0.0.1 33057 <nil> <nil>}
I0216 17:43:53.252909 305464 main.go:141] libmachine: About to run SSH command:
sudo hostname old-k8s-version-488384 && echo "old-k8s-version-488384" | sudo tee /etc/hostname
I0216 17:43:53.253474 305464 main.go:141] libmachine: Error dialing TCP: ssh: handshake failed: read tcp 127.0.0.1:53186->127.0.0.1:33057: read: connection reset by peer
I0216 17:43:56.441420 305464 main.go:141] libmachine: SSH cmd err, output: <nil>: old-k8s-version-488384
I0216 17:43:56.441537 305464 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-488384
I0216 17:43:56.469959 305464 main.go:141] libmachine: Using SSH client type: native
I0216 17:43:56.470435 305464 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3bf490] 0x3c1c00 <nil> [] 0s} 127.0.0.1 33057 <nil> <nil>}
I0216 17:43:56.470463 305464 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sold-k8s-version-488384' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 old-k8s-version-488384/g' /etc/hosts;
else
echo '127.0.1.1 old-k8s-version-488384' | sudo tee -a /etc/hosts;
fi
fi
I0216 17:43:56.633059 305464 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0216 17:43:56.633097 305464 ubuntu.go:175] set auth options {CertDir:/home/jenkins/minikube-integration/17936-2208/.minikube CaCertPath:/home/jenkins/minikube-integration/17936-2208/.minikube/certs/ca.pem CaPrivateKeyPath:/home/jenkins/minikube-integration/17936-2208/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/home/jenkins/minikube-integration/17936-2208/.minikube/machines/server.pem ServerKeyPath:/home/jenkins/minikube-integration/17936-2208/.minikube/machines/server-key.pem ClientKeyPath:/home/jenkins/minikube-integration/17936-2208/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/home/jenkins/minikube-integration/17936-2208/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/home/jenkins/minikube-integration/17936-2208/.minikube}
I0216 17:43:56.633154 305464 ubuntu.go:177] setting up certificates
I0216 17:43:56.633164 305464 provision.go:83] configureAuth start
I0216 17:43:56.633314 305464 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-488384
I0216 17:43:56.659937 305464 provision.go:138] copyHostCerts
I0216 17:43:56.660008 305464 exec_runner.go:144] found /home/jenkins/minikube-integration/17936-2208/.minikube/ca.pem, removing ...
I0216 17:43:56.660022 305464 exec_runner.go:203] rm: /home/jenkins/minikube-integration/17936-2208/.minikube/ca.pem
I0216 17:43:56.660108 305464 exec_runner.go:151] cp: /home/jenkins/minikube-integration/17936-2208/.minikube/certs/ca.pem --> /home/jenkins/minikube-integration/17936-2208/.minikube/ca.pem (1078 bytes)
I0216 17:43:56.660210 305464 exec_runner.go:144] found /home/jenkins/minikube-integration/17936-2208/.minikube/cert.pem, removing ...
I0216 17:43:56.660224 305464 exec_runner.go:203] rm: /home/jenkins/minikube-integration/17936-2208/.minikube/cert.pem
I0216 17:43:56.660254 305464 exec_runner.go:151] cp: /home/jenkins/minikube-integration/17936-2208/.minikube/certs/cert.pem --> /home/jenkins/minikube-integration/17936-2208/.minikube/cert.pem (1123 bytes)
I0216 17:43:56.660311 305464 exec_runner.go:144] found /home/jenkins/minikube-integration/17936-2208/.minikube/key.pem, removing ...
I0216 17:43:56.660321 305464 exec_runner.go:203] rm: /home/jenkins/minikube-integration/17936-2208/.minikube/key.pem
I0216 17:43:56.660346 305464 exec_runner.go:151] cp: /home/jenkins/minikube-integration/17936-2208/.minikube/certs/key.pem --> /home/jenkins/minikube-integration/17936-2208/.minikube/key.pem (1675 bytes)
I0216 17:43:56.660412 305464 provision.go:112] generating server cert: /home/jenkins/minikube-integration/17936-2208/.minikube/machines/server.pem ca-key=/home/jenkins/minikube-integration/17936-2208/.minikube/certs/ca.pem private-key=/home/jenkins/minikube-integration/17936-2208/.minikube/certs/ca-key.pem org=jenkins.old-k8s-version-488384 san=[192.168.67.2 127.0.0.1 localhost 127.0.0.1 minikube old-k8s-version-488384]
I0216 17:43:56.901566 305464 provision.go:172] copyRemoteCerts
I0216 17:43:56.901648 305464 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0216 17:43:56.901701 305464 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-488384
I0216 17:43:56.921863 305464 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33057 SSHKeyPath:/home/jenkins/minikube-integration/17936-2208/.minikube/machines/old-k8s-version-488384/id_rsa Username:docker}
I0216 17:43:57.027177 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1679 bytes)
I0216 17:43:57.058206 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1078 bytes)
I0216 17:43:57.086375 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/machines/server.pem --> /etc/docker/server.pem (1241 bytes)
I0216 17:43:57.114924 305464 provision.go:86] duration metric: configureAuth took 481.739759ms
I0216 17:43:57.114996 305464 ubuntu.go:193] setting minikube options for container-runtime
I0216 17:43:57.115231 305464 config.go:182] Loaded profile config "old-k8s-version-488384": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.16.0
I0216 17:43:57.115311 305464 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-488384
I0216 17:43:57.134286 305464 main.go:141] libmachine: Using SSH client type: native
I0216 17:43:57.134722 305464 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3bf490] 0x3c1c00 <nil> [] 0s} 127.0.0.1 33057 <nil> <nil>}
I0216 17:43:57.134735 305464 main.go:141] libmachine: About to run SSH command:
df --output=fstype / | tail -n 1
I0216 17:43:57.282315 305464 main.go:141] libmachine: SSH cmd err, output: <nil>: overlay
I0216 17:43:57.282335 305464 ubuntu.go:71] root file system type: overlay
I0216 17:43:57.282451 305464 provision.go:309] Updating docker unit: /lib/systemd/system/docker.service ...
I0216 17:43:57.282518 305464 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-488384
I0216 17:43:57.303772 305464 main.go:141] libmachine: Using SSH client type: native
I0216 17:43:57.304206 305464 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3bf490] 0x3c1c00 <nil> [] 0s} 127.0.0.1 33057 <nil> <nil>}
I0216 17:43:57.304285 305464 main.go:141] libmachine: About to run SSH command:
sudo mkdir -p /lib/systemd/system && printf %s "[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP \$MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service.new
I0216 17:43:57.474048 305464 main.go:141] libmachine: SSH cmd err, output: <nil>: [Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
I0216 17:43:57.474203 305464 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-488384
I0216 17:43:57.495804 305464 main.go:141] libmachine: Using SSH client type: native
I0216 17:43:57.496220 305464 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x3bf490] 0x3c1c00 <nil> [] 0s} 127.0.0.1 33057 <nil> <nil>}
I0216 17:43:57.496238 305464 main.go:141] libmachine: About to run SSH command:
sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
I0216 17:43:58.414273 305464 main.go:141] libmachine: SSH cmd err, output: <nil>: --- /lib/systemd/system/docker.service 2024-02-06 21:12:51.000000000 +0000
+++ /lib/systemd/system/docker.service.new 2024-02-16 17:43:57.464880626 +0000
@@ -1,46 +1,49 @@
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
-After=network-online.target docker.socket firewalld.service containerd.service time-set.target
-Wants=network-online.target containerd.service
+BindsTo=containerd.service
+After=network-online.target firewalld.service containerd.service
+Wants=network-online.target
Requires=docker.socket
+StartLimitBurst=3
+StartLimitIntervalSec=60
[Service]
Type=notify
-# the default is not to use systemd for cgroups because the delegate issues still
-# exists and systemd currently does not support the cgroup feature set required
-# for containers run by docker
-ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
-ExecReload=/bin/kill -s HUP $MAINPID
-TimeoutStartSec=0
-RestartSec=2
-Restart=always
+Restart=on-failure
-# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
-# Both the old, and new location are accepted by systemd 229 and up, so using the old location
-# to make them work for either version of systemd.
-StartLimitBurst=3
-# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
-# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
-# this option work for either version of systemd.
-StartLimitInterval=60s
+
+# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
+# The base configuration already specifies an 'ExecStart=...' command. The first directive
+# here is to clear out that command inherited from the base configuration. Without this,
+# the command from the base configuration and the command specified here are treated as
+# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
+# will catch this invalid input and refuse to start the service with an error like:
+# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
+
+# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
+# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
+ExecStart=
+ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
+ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
+LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
-# Comment TasksMax if your systemd version does not support it.
-# Only systemd 226 and above support this option.
+# Uncomment TasksMax if your systemd version supports it.
+# Only systemd 226 and above support this version.
TasksMax=infinity
+TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
-OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker
I0216 17:43:58.414304 305464 machine.go:91] provisioned docker machine in 5.18546255s
I0216 17:43:58.414317 305464 client.go:171] LocalClient.Create took 12.183614086s
I0216 17:43:58.414331 305464 start.go:167] duration metric: libmachine.API.Create for "old-k8s-version-488384" took 12.18366463s
I0216 17:43:58.414357 305464 start.go:300] post-start starting for "old-k8s-version-488384" (driver="docker")
I0216 17:43:58.414373 305464 start.go:329] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0216 17:43:58.414440 305464 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0216 17:43:58.414486 305464 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-488384
I0216 17:43:58.431251 305464 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33057 SSHKeyPath:/home/jenkins/minikube-integration/17936-2208/.minikube/machines/old-k8s-version-488384/id_rsa Username:docker}
I0216 17:43:58.534236 305464 ssh_runner.go:195] Run: cat /etc/os-release
I0216 17:43:58.537406 305464 main.go:141] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0216 17:43:58.537448 305464 main.go:141] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0216 17:43:58.537480 305464 main.go:141] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0216 17:43:58.537494 305464 info.go:137] Remote host: Ubuntu 22.04.3 LTS
I0216 17:43:58.537505 305464 filesync.go:126] Scanning /home/jenkins/minikube-integration/17936-2208/.minikube/addons for local assets ...
I0216 17:43:58.537571 305464 filesync.go:126] Scanning /home/jenkins/minikube-integration/17936-2208/.minikube/files for local assets ...
I0216 17:43:58.537656 305464 filesync.go:149] local asset: /home/jenkins/minikube-integration/17936-2208/.minikube/files/etc/ssl/certs/75132.pem -> 75132.pem in /etc/ssl/certs
I0216 17:43:58.537765 305464 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0216 17:43:58.546207 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/files/etc/ssl/certs/75132.pem --> /etc/ssl/certs/75132.pem (1708 bytes)
I0216 17:43:58.571169 305464 start.go:303] post-start completed in 156.797543ms
I0216 17:43:58.571534 305464 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-488384
I0216 17:43:58.602067 305464 profile.go:148] Saving config to /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/config.json ...
I0216 17:43:58.602353 305464 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0216 17:43:58.602418 305464 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-488384
I0216 17:43:58.623992 305464 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33057 SSHKeyPath:/home/jenkins/minikube-integration/17936-2208/.minikube/machines/old-k8s-version-488384/id_rsa Username:docker}
I0216 17:43:58.722718 305464 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0216 17:43:58.727943 305464 start.go:128] duration metric: createHost completed in 12.499618522s
I0216 17:43:58.727969 305464 start.go:83] releasing machines lock for "old-k8s-version-488384", held for 12.499748574s
I0216 17:43:58.728038 305464 cli_runner.go:164] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" old-k8s-version-488384
I0216 17:43:58.755448 305464 ssh_runner.go:195] Run: cat /version.json
I0216 17:43:58.755670 305464 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-488384
I0216 17:43:58.755620 305464 ssh_runner.go:195] Run: curl -sS -m 2 https://registry.k8s.io/
I0216 17:43:58.755968 305464 cli_runner.go:164] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" old-k8s-version-488384
I0216 17:43:58.790423 305464 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33057 SSHKeyPath:/home/jenkins/minikube-integration/17936-2208/.minikube/machines/old-k8s-version-488384/id_rsa Username:docker}
I0216 17:43:58.801541 305464 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:33057 SSHKeyPath:/home/jenkins/minikube-integration/17936-2208/.minikube/machines/old-k8s-version-488384/id_rsa Username:docker}
I0216 17:43:58.892105 305464 ssh_runner.go:195] Run: systemctl --version
I0216 17:43:59.027395 305464 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
I0216 17:43:59.031600 305464 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *loopback.conf* -not -name *.mk_disabled -exec sh -c "grep -q loopback {} && ( grep -q name {} || sudo sed -i '/"type": "loopback"/i \ \ \ \ "name": "loopback",' {} ) && sudo sed -i 's|"cniVersion": ".*"|"cniVersion": "1.0.0"|g' {}" ;
I0216 17:43:59.056897 305464 cni.go:230] loopback cni configuration patched: "/etc/cni/net.d/*loopback.conf*" found
I0216 17:43:59.056976 305464 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *bridge* -not -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e '/"dst": ".*:.*"/d' -e 's|^(.*)"dst": (.*)[,*]$|\1"dst": \2|g' -e '/"subnet": ".*:.*"/d' -e 's|^(.*)"subnet": ".*"(.*)[,*]$|\1"subnet": "10.244.0.0/16"\2|g' {}" ;
I0216 17:43:59.074130 305464 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f -name *podman* -not -name *.mk_disabled -printf "%p, " -exec sh -c "sudo sed -i -r -e 's|^(.*)"subnet": ".*"(.*)$|\1"subnet": "10.244.0.0/16"\2|g' -e 's|^(.*)"gateway": ".*"(.*)$|\1"gateway": "10.244.0.1"\2|g' {}" ;
I0216 17:43:59.093821 305464 cni.go:308] configured [/etc/cni/net.d/100-crio-bridge.conf, /etc/cni/net.d/87-podman-bridge.conflist] bridge cni config(s)
I0216 17:43:59.093884 305464 start.go:475] detecting cgroup driver to use...
I0216 17:43:59.093930 305464 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0216 17:43:59.094054 305464 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///run/containerd/containerd.sock
" | sudo tee /etc/crictl.yaml"
I0216 17:43:59.112256 305464 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)sandbox_image = .*$|\1sandbox_image = "registry.k8s.io/pause:3.1"|' /etc/containerd/config.toml"
I0216 17:43:59.123216 305464 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)restrict_oom_score_adj = .*$|\1restrict_oom_score_adj = false|' /etc/containerd/config.toml"
I0216 17:43:59.133463 305464 containerd.go:146] configuring containerd to use "cgroupfs" as cgroup driver...
I0216 17:43:59.133529 305464 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)SystemdCgroup = .*$|\1SystemdCgroup = false|g' /etc/containerd/config.toml"
I0216 17:43:59.146448 305464 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runtime.v1.linux"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0216 17:43:59.157027 305464 ssh_runner.go:195] Run: sh -c "sudo sed -i '/systemd_cgroup/d' /etc/containerd/config.toml"
I0216 17:43:59.167090 305464 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|"io.containerd.runc.v1"|"io.containerd.runc.v2"|g' /etc/containerd/config.toml"
I0216 17:43:59.177619 305464 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0216 17:43:59.194612 305464 ssh_runner.go:195] Run: sh -c "sudo sed -i -r 's|^( *)conf_dir = .*$|\1conf_dir = "/etc/cni/net.d"|g' /etc/containerd/config.toml"
I0216 17:43:59.205062 305464 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0216 17:43:59.214886 305464 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0216 17:43:59.223572 305464 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0216 17:43:59.308385 305464 ssh_runner.go:195] Run: sudo systemctl restart containerd
I0216 17:43:59.405630 305464 start.go:475] detecting cgroup driver to use...
I0216 17:43:59.405675 305464 detect.go:196] detected "cgroupfs" cgroup driver on host os
I0216 17:43:59.405725 305464 ssh_runner.go:195] Run: sudo systemctl cat docker.service
I0216 17:43:59.420521 305464 cruntime.go:279] skipping containerd shutdown because we are bound to it
I0216 17:43:59.420597 305464 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service crio
I0216 17:43:59.434085 305464 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///var/run/dockershim.sock
" | sudo tee /etc/crictl.yaml"
I0216 17:43:59.456016 305464 ssh_runner.go:195] Run: which cri-dockerd
I0216 17:43:59.460151 305464 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/cri-docker.service.d
I0216 17:43:59.468733 305464 ssh_runner.go:362] scp memory --> /etc/systemd/system/cri-docker.service.d/10-cni.conf (189 bytes)
I0216 17:43:59.486894 305464 ssh_runner.go:195] Run: sudo systemctl unmask docker.service
I0216 17:43:59.605206 305464 ssh_runner.go:195] Run: sudo systemctl enable docker.socket
I0216 17:43:59.706569 305464 docker.go:574] configuring docker to use "cgroupfs" as cgroup driver...
I0216 17:43:59.706797 305464 ssh_runner.go:362] scp memory --> /etc/docker/daemon.json (130 bytes)
I0216 17:43:59.732730 305464 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0216 17:43:59.865734 305464 ssh_runner.go:195] Run: sudo systemctl restart docker
I0216 17:44:00.353193 305464 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0216 17:44:00.395744 305464 ssh_runner.go:195] Run: docker version --format {{.Server.Version}}
I0216 17:44:00.455023 305464 out.go:204] * Preparing Kubernetes v1.16.0 on Docker 25.0.3 ...
I0216 17:44:00.455143 305464 cli_runner.go:164] Run: docker network inspect old-k8s-version-488384 --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0216 17:44:00.480695 305464 ssh_runner.go:195] Run: grep 192.168.67.1 host.minikube.internal$ /etc/hosts
I0216 17:44:00.487221 305464 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.67.1 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0216 17:44:00.502829 305464 preload.go:132] Checking if preload exists for k8s version v1.16.0 and runtime docker
I0216 17:44:00.502916 305464 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0216 17:44:00.541578 305464 docker.go:685] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0216 17:44:00.541601 305464 docker.go:691] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0216 17:44:00.541657 305464 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0216 17:44:00.556984 305464 ssh_runner.go:195] Run: which lz4
I0216 17:44:00.563250 305464 ssh_runner.go:195] Run: stat -c "%s %y" /preloaded.tar.lz4
I0216 17:44:00.568895 305464 ssh_runner.go:352] existence check for /preloaded.tar.lz4: stat -c "%s %y" /preloaded.tar.lz4: Process exited with status 1
stdout:
stderr:
stat: cannot statx '/preloaded.tar.lz4': No such file or directory
I0216 17:44:00.568929 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.16.0-docker-overlay2-arm64.tar.lz4 --> /preloaded.tar.lz4 (394173841 bytes)
I0216 17:44:02.782914 305464 docker.go:649] Took 2.219703 seconds to copy over tarball
I0216 17:44:02.783004 305464 ssh_runner.go:195] Run: sudo tar --xattrs --xattrs-include security.capability -I lz4 -C /var -xf /preloaded.tar.lz4
I0216 17:44:05.379588 305464 ssh_runner.go:235] Completed: sudo tar --xattrs --xattrs-include security.capability -I lz4 -C /var -xf /preloaded.tar.lz4: (2.596553504s)
I0216 17:44:05.379658 305464 ssh_runner.go:146] rm: /preloaded.tar.lz4
I0216 17:44:05.449612 305464 ssh_runner.go:195] Run: sudo cat /var/lib/docker/image/overlay2/repositories.json
I0216 17:44:05.458377 305464 ssh_runner.go:362] scp memory --> /var/lib/docker/image/overlay2/repositories.json (2499 bytes)
I0216 17:44:05.476927 305464 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0216 17:44:05.570190 305464 ssh_runner.go:195] Run: sudo systemctl restart docker
I0216 17:44:06.850970 305464 ssh_runner.go:235] Completed: sudo systemctl restart docker: (1.280739616s)
I0216 17:44:06.851073 305464 ssh_runner.go:195] Run: docker images --format {{.Repository}}:{{.Tag}}
I0216 17:44:06.878741 305464 docker.go:685] Got preloaded images: -- stdout --
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/kube-controller-manager:v1.16.0
k8s.gcr.io/kube-apiserver:v1.16.0
k8s.gcr.io/kube-proxy:v1.16.0
k8s.gcr.io/kube-scheduler:v1.16.0
k8s.gcr.io/etcd:3.3.15-0
k8s.gcr.io/coredns:1.6.2
k8s.gcr.io/pause:3.1
-- /stdout --
I0216 17:44:06.878758 305464 docker.go:691] registry.k8s.io/kube-apiserver:v1.16.0 wasn't preloaded
I0216 17:44:06.878767 305464 cache_images.go:88] LoadImages start: [registry.k8s.io/kube-apiserver:v1.16.0 registry.k8s.io/kube-controller-manager:v1.16.0 registry.k8s.io/kube-scheduler:v1.16.0 registry.k8s.io/kube-proxy:v1.16.0 registry.k8s.io/pause:3.1 registry.k8s.io/etcd:3.3.15-0 registry.k8s.io/coredns:1.6.2 gcr.io/k8s-minikube/storage-provisioner:v5]
I0216 17:44:06.880710 305464 image.go:134] retrieving image: gcr.io/k8s-minikube/storage-provisioner:v5
I0216 17:44:06.880806 305464 image.go:134] retrieving image: registry.k8s.io/kube-controller-manager:v1.16.0
I0216 17:44:06.880924 305464 image.go:134] retrieving image: registry.k8s.io/coredns:1.6.2
I0216 17:44:06.880950 305464 image.go:134] retrieving image: registry.k8s.io/kube-apiserver:v1.16.0
I0216 17:44:06.881003 305464 image.go:134] retrieving image: registry.k8s.io/pause:3.1
I0216 17:44:06.881092 305464 image.go:134] retrieving image: registry.k8s.io/etcd:3.3.15-0
I0216 17:44:06.881137 305464 image.go:134] retrieving image: registry.k8s.io/kube-proxy:v1.16.0
I0216 17:44:06.881897 305464 image.go:134] retrieving image: registry.k8s.io/kube-scheduler:v1.16.0
I0216 17:44:06.882487 305464 image.go:177] daemon lookup for registry.k8s.io/kube-proxy:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-proxy:v1.16.0
I0216 17:44:06.882936 305464 image.go:177] daemon lookup for registry.k8s.io/kube-apiserver:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-apiserver:v1.16.0
I0216 17:44:06.883097 305464 image.go:177] daemon lookup for registry.k8s.io/coredns:1.6.2: Error response from daemon: No such image: registry.k8s.io/coredns:1.6.2
I0216 17:44:06.883213 305464 image.go:177] daemon lookup for registry.k8s.io/kube-controller-manager:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-controller-manager:v1.16.0
I0216 17:44:06.883337 305464 image.go:177] daemon lookup for gcr.io/k8s-minikube/storage-provisioner:v5: Error response from daemon: No such image: gcr.io/k8s-minikube/storage-provisioner:v5
I0216 17:44:06.883642 305464 image.go:177] daemon lookup for registry.k8s.io/pause:3.1: Error response from daemon: No such image: registry.k8s.io/pause:3.1
I0216 17:44:06.883907 305464 image.go:177] daemon lookup for registry.k8s.io/etcd:3.3.15-0: Error response from daemon: No such image: registry.k8s.io/etcd:3.3.15-0
I0216 17:44:06.885129 305464 image.go:177] daemon lookup for registry.k8s.io/kube-scheduler:v1.16.0: Error response from daemon: No such image: registry.k8s.io/kube-scheduler:v1.16.0
W0216 17:44:07.213858 305464 image.go:265] image registry.k8s.io/pause:3.1 arch mismatch: want arm64 got amd64. fixing
I0216 17:44:07.214009 305464 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/pause:3.1
W0216 17:44:07.230578 305464 image.go:265] image registry.k8s.io/kube-scheduler:v1.16.0 arch mismatch: want arm64 got amd64. fixing
I0216 17:44:07.230794 305464 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-scheduler:v1.16.0
I0216 17:44:07.233459 305464 cache_images.go:116] "registry.k8s.io/pause:3.1" needs transfer: "registry.k8s.io/pause:3.1" does not exist at hash "8057e0500773a37cde2cff041eb13ebd68c748419a2fbfd1dfb5bf38696cc8e5" in container runtime
I0216 17:44:07.233547 305464 docker.go:337] Removing image: registry.k8s.io/pause:3.1
I0216 17:44:07.233607 305464 ssh_runner.go:195] Run: docker rmi registry.k8s.io/pause:3.1
W0216 17:44:07.241230 305464 image.go:265] image registry.k8s.io/kube-apiserver:v1.16.0 arch mismatch: want arm64 got amd64. fixing
I0216 17:44:07.241420 305464 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-apiserver:v1.16.0
W0216 17:44:07.260438 305464 image.go:265] image registry.k8s.io/coredns:1.6.2 arch mismatch: want arm64 got amd64. fixing
I0216 17:44:07.260664 305464 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/coredns:1.6.2
I0216 17:44:07.263689 305464 cache_images.go:116] "registry.k8s.io/kube-scheduler:v1.16.0" needs transfer: "registry.k8s.io/kube-scheduler:v1.16.0" does not exist at hash "e82518cbd8204462b7b3756330f327ee6de72bbb84aaebc4c8cadf77c821a661" in container runtime
I0216 17:44:07.263751 305464 docker.go:337] Removing image: registry.k8s.io/kube-scheduler:v1.16.0
I0216 17:44:07.263809 305464 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-scheduler:v1.16.0
W0216 17:44:07.266406 305464 image.go:265] image registry.k8s.io/kube-controller-manager:v1.16.0 arch mismatch: want arm64 got amd64. fixing
I0216 17:44:07.266622 305464 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-controller-manager:v1.16.0
W0216 17:44:07.267541 305464 image.go:265] image registry.k8s.io/etcd:3.3.15-0 arch mismatch: want arm64 got amd64. fixing
I0216 17:44:07.267791 305464 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/etcd:3.3.15-0
W0216 17:44:07.275716 305464 image.go:265] image registry.k8s.io/kube-proxy:v1.16.0 arch mismatch: want arm64 got amd64. fixing
I0216 17:44:07.275951 305464 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} registry.k8s.io/kube-proxy:v1.16.0
I0216 17:44:07.405266 305464 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/17936-2208/.minikube/cache/images/arm64/registry.k8s.io/pause_3.1
I0216 17:44:07.405312 305464 cache_images.go:116] "registry.k8s.io/kube-apiserver:v1.16.0" needs transfer: "registry.k8s.io/kube-apiserver:v1.16.0" does not exist at hash "06c3d6081b24d1d3f9c703ae2e40666f3237db9490060a03c4b29894a78205ef" in container runtime
I0216 17:44:07.405336 305464 docker.go:337] Removing image: registry.k8s.io/kube-apiserver:v1.16.0
I0216 17:44:07.405377 305464 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-apiserver:v1.16.0
I0216 17:44:07.407371 305464 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/17936-2208/.minikube/cache/images/arm64/registry.k8s.io/kube-scheduler_v1.16.0
I0216 17:44:07.407412 305464 cache_images.go:116] "registry.k8s.io/kube-controller-manager:v1.16.0" needs transfer: "registry.k8s.io/kube-controller-manager:v1.16.0" does not exist at hash "5f93833eff730f6c51ed0232bb218db5ab7bbb05ed0d460c4678d8b433670640" in container runtime
I0216 17:44:07.407444 305464 docker.go:337] Removing image: registry.k8s.io/kube-controller-manager:v1.16.0
I0216 17:44:07.407477 305464 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-controller-manager:v1.16.0
I0216 17:44:07.407521 305464 cache_images.go:116] "registry.k8s.io/etcd:3.3.15-0" needs transfer: "registry.k8s.io/etcd:3.3.15-0" does not exist at hash "3f4e1b5a89fe11634ed042397d01167d866dfa3225cfed8279f54ec7f8f58486" in container runtime
I0216 17:44:07.407534 305464 docker.go:337] Removing image: registry.k8s.io/etcd:3.3.15-0
I0216 17:44:07.407553 305464 ssh_runner.go:195] Run: docker rmi registry.k8s.io/etcd:3.3.15-0
I0216 17:44:07.407857 305464 cache_images.go:116] "registry.k8s.io/kube-proxy:v1.16.0" needs transfer: "registry.k8s.io/kube-proxy:v1.16.0" does not exist at hash "553e5791dff2eebc7969b9df892ad18a487fcfa425e098ed3059173e36d98f72" in container runtime
I0216 17:44:07.407878 305464 docker.go:337] Removing image: registry.k8s.io/kube-proxy:v1.16.0
I0216 17:44:07.407908 305464 ssh_runner.go:195] Run: docker rmi registry.k8s.io/kube-proxy:v1.16.0
I0216 17:44:07.410985 305464 cache_images.go:116] "registry.k8s.io/coredns:1.6.2" needs transfer: "registry.k8s.io/coredns:1.6.2" does not exist at hash "051b2962d7b329402cf101d688a2de7bc400efea9dd4de77745af5d77489a847" in container runtime
I0216 17:44:07.411043 305464 docker.go:337] Removing image: registry.k8s.io/coredns:1.6.2
I0216 17:44:07.411108 305464 ssh_runner.go:195] Run: docker rmi registry.k8s.io/coredns:1.6.2
I0216 17:44:07.435743 305464 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/17936-2208/.minikube/cache/images/arm64/registry.k8s.io/kube-apiserver_v1.16.0
I0216 17:44:07.468527 305464 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/17936-2208/.minikube/cache/images/arm64/registry.k8s.io/etcd_3.3.15-0
I0216 17:44:07.468627 305464 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/17936-2208/.minikube/cache/images/arm64/registry.k8s.io/kube-controller-manager_v1.16.0
I0216 17:44:07.469573 305464 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/17936-2208/.minikube/cache/images/arm64/registry.k8s.io/coredns_1.6.2
I0216 17:44:07.469595 305464 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/17936-2208/.minikube/cache/images/arm64/registry.k8s.io/kube-proxy_v1.16.0
W0216 17:44:07.507667 305464 image.go:265] image gcr.io/k8s-minikube/storage-provisioner:v5 arch mismatch: want arm64 got amd64. fixing
I0216 17:44:07.507815 305464 ssh_runner.go:195] Run: docker image inspect --format {{.Id}} gcr.io/k8s-minikube/storage-provisioner:v5
I0216 17:44:07.525954 305464 cache_images.go:116] "gcr.io/k8s-minikube/storage-provisioner:v5" needs transfer: "gcr.io/k8s-minikube/storage-provisioner:v5" does not exist at hash "66749159455b3f08c8318fe0233122f54d0f5889f9c5fdfb73c3fd9d99895b51" in container runtime
I0216 17:44:07.526050 305464 docker.go:337] Removing image: gcr.io/k8s-minikube/storage-provisioner:v5
I0216 17:44:07.526125 305464 ssh_runner.go:195] Run: docker rmi gcr.io/k8s-minikube/storage-provisioner:v5
I0216 17:44:07.555791 305464 cache_images.go:286] Loading image from: /home/jenkins/minikube-integration/17936-2208/.minikube/cache/images/arm64/gcr.io/k8s-minikube/storage-provisioner_v5
I0216 17:44:07.555958 305464 ssh_runner.go:195] Run: stat -c "%s %y" /var/lib/minikube/images/storage-provisioner_v5
I0216 17:44:07.559524 305464 ssh_runner.go:352] existence check for /var/lib/minikube/images/storage-provisioner_v5: stat -c "%s %y" /var/lib/minikube/images/storage-provisioner_v5: Process exited with status 1
stdout:
stderr:
stat: cannot statx '/var/lib/minikube/images/storage-provisioner_v5': No such file or directory
I0216 17:44:07.559561 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/cache/images/arm64/gcr.io/k8s-minikube/storage-provisioner_v5 --> /var/lib/minikube/images/storage-provisioner_v5 (8035840 bytes)
I0216 17:44:07.648083 305464 docker.go:304] Loading image: /var/lib/minikube/images/storage-provisioner_v5
I0216 17:44:07.648148 305464 ssh_runner.go:195] Run: /bin/bash -c "sudo cat /var/lib/minikube/images/storage-provisioner_v5 | docker load"
I0216 17:44:07.968680 305464 cache_images.go:315] Transferred and loaded /home/jenkins/minikube-integration/17936-2208/.minikube/cache/images/arm64/gcr.io/k8s-minikube/storage-provisioner_v5 from cache
I0216 17:44:07.968745 305464 cache_images.go:92] LoadImages completed in 1.089939898s
W0216 17:44:07.968812 305464 out.go:239] X Unable to load cached images: loading cached images: stat /home/jenkins/minikube-integration/17936-2208/.minikube/cache/images/arm64/registry.k8s.io/pause_3.1: no such file or directory
X Unable to load cached images: loading cached images: stat /home/jenkins/minikube-integration/17936-2208/.minikube/cache/images/arm64/registry.k8s.io/pause_3.1: no such file or directory
I0216 17:44:07.968875 305464 ssh_runner.go:195] Run: docker info --format {{.CgroupDriver}}
I0216 17:44:08.026505 305464 cni.go:84] Creating CNI manager for ""
I0216 17:44:08.026531 305464 cni.go:162] CNI unnecessary in this configuration, recommending no CNI
I0216 17:44:08.026547 305464 kubeadm.go:87] Using pod CIDR: 10.244.0.0/16
I0216 17:44:08.026566 305464 kubeadm.go:176] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.67.2 APIServerPort:8443 KubernetesVersion:v1.16.0 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:old-k8s-version-488384 NodeName:old-k8s-version-488384 DNSDomain:cluster.local CRISocket:/var/run/dockershim.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.67.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.67.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticP
odPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[hairpinMode:hairpin-veth runtimeRequestTimeout:15m] PrependCriSocketUnix:false}
I0216 17:44:08.026707 305464 kubeadm.go:181] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.67.2
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: "old-k8s-version-488384"
kubeletExtraArgs:
node-ip: 192.168.67.2
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.67.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: old-k8s-version-488384
controlPlaneEndpoint: control-plane.minikube.internal:8443
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
listen-metrics-urls: http://127.0.0.1:2381,http://192.168.67.2:2381
kubernetesVersion: v1.16.0
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0216 17:44:08.026774 305464 kubeadm.go:976] kubelet [Unit]
Wants=docker.socket
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.16.0/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=old-k8s-version-488384 --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.67.2
[Install]
config:
{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-488384 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:}
I0216 17:44:08.026844 305464 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.16.0
I0216 17:44:08.036111 305464 binaries.go:44] Found k8s binaries, skipping transfer
I0216 17:44:08.036186 305464 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0216 17:44:08.045523 305464 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (348 bytes)
I0216 17:44:08.064272 305464 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0216 17:44:08.083503 305464 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2174 bytes)
I0216 17:44:08.102845 305464 ssh_runner.go:195] Run: grep 192.168.67.2 control-plane.minikube.internal$ /etc/hosts
I0216 17:44:08.106515 305464 ssh_runner.go:195] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.67.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0216 17:44:08.117361 305464 certs.go:56] Setting up /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384 for IP: 192.168.67.2
I0216 17:44:08.117431 305464 certs.go:190] acquiring lock for shared ca certs: {Name:mkc4dfb4b2b1da0d6a80fb9567025307b764443b Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:44:08.117564 305464 certs.go:199] skipping minikubeCA CA generation: /home/jenkins/minikube-integration/17936-2208/.minikube/ca.key
I0216 17:44:08.117613 305464 certs.go:199] skipping proxyClientCA CA generation: /home/jenkins/minikube-integration/17936-2208/.minikube/proxy-client-ca.key
I0216 17:44:08.117666 305464 certs.go:319] generating minikube-user signed cert: /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/client.key
I0216 17:44:08.117681 305464 crypto.go:68] Generating cert /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/client.crt with IP's: []
I0216 17:44:08.490558 305464 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/client.crt ...
I0216 17:44:08.490590 305464 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/client.crt: {Name:mk6be765281296a271bf945592d12193d13e3a57 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:44:08.490782 305464 crypto.go:164] Writing key to /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/client.key ...
I0216 17:44:08.490795 305464 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/client.key: {Name:mkb9946397b6b825a55f47de800205d5cf9aacff Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:44:08.490881 305464 certs.go:319] generating minikube signed cert: /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/apiserver.key.c7fa3a9e
I0216 17:44:08.490904 305464 crypto.go:68] Generating cert /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/apiserver.crt.c7fa3a9e with IP's: [192.168.67.2 10.96.0.1 127.0.0.1 10.0.0.1]
I0216 17:44:08.899550 305464 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/apiserver.crt.c7fa3a9e ...
I0216 17:44:08.899578 305464 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/apiserver.crt.c7fa3a9e: {Name:mkfcbc3980467b24265bec39c546f72a3c038e54 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:44:08.900118 305464 crypto.go:164] Writing key to /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/apiserver.key.c7fa3a9e ...
I0216 17:44:08.900136 305464 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/apiserver.key.c7fa3a9e: {Name:mk2cb1c7d0f1444f376d1baa587b07633a0736ea Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:44:08.900226 305464 certs.go:337] copying /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/apiserver.crt.c7fa3a9e -> /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/apiserver.crt
I0216 17:44:08.900306 305464 certs.go:341] copying /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/apiserver.key.c7fa3a9e -> /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/apiserver.key
I0216 17:44:08.900363 305464 certs.go:319] generating aggregator signed cert: /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/proxy-client.key
I0216 17:44:08.900381 305464 crypto.go:68] Generating cert /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/proxy-client.crt with IP's: []
I0216 17:44:09.182378 305464 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/proxy-client.crt ...
I0216 17:44:09.182408 305464 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/proxy-client.crt: {Name:mk1637ac314524cbe2b7f2a1e52f306bab041e7e Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:44:09.182585 305464 crypto.go:164] Writing key to /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/proxy-client.key ...
I0216 17:44:09.182600 305464 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/proxy-client.key: {Name:mkec021588dc3dd0f104d522dedf7c0eb10e8606 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0216 17:44:09.183230 305464 certs.go:437] found cert: /home/jenkins/minikube-integration/17936-2208/.minikube/certs/home/jenkins/minikube-integration/17936-2208/.minikube/certs/7513.pem (1338 bytes)
W0216 17:44:09.183274 305464 certs.go:433] ignoring /home/jenkins/minikube-integration/17936-2208/.minikube/certs/home/jenkins/minikube-integration/17936-2208/.minikube/certs/7513_empty.pem, impossibly tiny 0 bytes
I0216 17:44:09.183287 305464 certs.go:437] found cert: /home/jenkins/minikube-integration/17936-2208/.minikube/certs/home/jenkins/minikube-integration/17936-2208/.minikube/certs/ca-key.pem (1679 bytes)
I0216 17:44:09.183314 305464 certs.go:437] found cert: /home/jenkins/minikube-integration/17936-2208/.minikube/certs/home/jenkins/minikube-integration/17936-2208/.minikube/certs/ca.pem (1078 bytes)
I0216 17:44:09.183347 305464 certs.go:437] found cert: /home/jenkins/minikube-integration/17936-2208/.minikube/certs/home/jenkins/minikube-integration/17936-2208/.minikube/certs/cert.pem (1123 bytes)
I0216 17:44:09.183378 305464 certs.go:437] found cert: /home/jenkins/minikube-integration/17936-2208/.minikube/certs/home/jenkins/minikube-integration/17936-2208/.minikube/certs/key.pem (1675 bytes)
I0216 17:44:09.183429 305464 certs.go:437] found cert: /home/jenkins/minikube-integration/17936-2208/.minikube/files/etc/ssl/certs/home/jenkins/minikube-integration/17936-2208/.minikube/files/etc/ssl/certs/75132.pem (1708 bytes)
I0216 17:44:09.184056 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1399 bytes)
I0216 17:44:09.209721 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1675 bytes)
I0216 17:44:09.235124 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0216 17:44:09.263947 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/profiles/old-k8s-version-488384/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1675 bytes)
I0216 17:44:09.292559 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0216 17:44:09.319693 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1679 bytes)
I0216 17:44:09.350278 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0216 17:44:09.379195 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1679 bytes)
I0216 17:44:09.404694 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0216 17:44:09.430101 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/certs/7513.pem --> /usr/share/ca-certificates/7513.pem (1338 bytes)
I0216 17:44:09.460339 305464 ssh_runner.go:362] scp /home/jenkins/minikube-integration/17936-2208/.minikube/files/etc/ssl/certs/75132.pem --> /usr/share/ca-certificates/75132.pem (1708 bytes)
I0216 17:44:09.485639 305464 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0216 17:44:09.504311 305464 ssh_runner.go:195] Run: openssl version
I0216 17:44:09.510095 305464 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/75132.pem && ln -fs /usr/share/ca-certificates/75132.pem /etc/ssl/certs/75132.pem"
I0216 17:44:09.519574 305464 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/75132.pem
I0216 17:44:09.523190 305464 certs.go:480] hashing: -rw-r--r-- 1 root root 1708 Feb 16 16:48 /usr/share/ca-certificates/75132.pem
I0216 17:44:09.523279 305464 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/75132.pem
I0216 17:44:09.530358 305464 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/75132.pem /etc/ssl/certs/3ec20f2e.0"
I0216 17:44:09.539765 305464 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0216 17:44:09.548940 305464 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0216 17:44:09.552401 305464 certs.go:480] hashing: -rw-r--r-- 1 root root 1111 Feb 16 16:42 /usr/share/ca-certificates/minikubeCA.pem
I0216 17:44:09.552466 305464 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0216 17:44:09.559783 305464 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0216 17:44:09.569557 305464 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/7513.pem && ln -fs /usr/share/ca-certificates/7513.pem /etc/ssl/certs/7513.pem"
I0216 17:44:09.579459 305464 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/7513.pem
I0216 17:44:09.583005 305464 certs.go:480] hashing: -rw-r--r-- 1 root root 1338 Feb 16 16:48 /usr/share/ca-certificates/7513.pem
I0216 17:44:09.583076 305464 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/7513.pem
I0216 17:44:09.590599 305464 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/7513.pem /etc/ssl/certs/51391683.0"
I0216 17:44:09.600265 305464 ssh_runner.go:195] Run: ls /var/lib/minikube/certs/etcd
I0216 17:44:09.604032 305464 certs.go:353] certs directory doesn't exist, likely first start: ls /var/lib/minikube/certs/etcd: Process exited with status 2
stdout:
stderr:
ls: cannot access '/var/lib/minikube/certs/etcd': No such file or directory
I0216 17:44:09.604109 305464 kubeadm.go:404] StartCluster: {Name:old-k8s-version-488384 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:true NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.16.0 ClusterName:old-k8s-version-488384 Namespace:default APIServerName:minikubeCA APISer
verNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.67.2 Port:8443 KubernetesVersion:v1.16.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:fals
e DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0216 17:44:09.604236 305464 ssh_runner.go:195] Run: docker ps --filter status=paused --filter=name=k8s_.*_(kube-system)_ --format={{.ID}}
I0216 17:44:09.621173 305464 ssh_runner.go:195] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0216 17:44:09.630943 305464 ssh_runner.go:195] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0216 17:44:09.640105 305464 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0216 17:44:09.640173 305464 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0216 17:44:09.649747 305464 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0216 17:44:09.649793 305464 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0216 17:44:09.878886 305464 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0216 17:44:09.938656 305464 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
I0216 17:44:09.938963 305464 kubeadm.go:322] [WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1053-aws\n", err: exit status 1
I0216 17:44:10.028999 305464 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0216 17:48:18.580985 305464 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0216 17:48:18.581096 305464 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
I0216 17:48:18.584855 305464 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0216 17:48:18.584922 305464 kubeadm.go:322] [preflight] Running pre-flight checks
I0216 17:48:18.585157 305464 kubeadm.go:322] [preflight] The system verification failed. Printing the output from the verification:
I0216 17:48:18.585241 305464 kubeadm.go:322] [0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1053-aws[0m
I0216 17:48:18.585302 305464 kubeadm.go:322] [0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
I0216 17:48:18.585339 305464 kubeadm.go:322] [0;37mOS[0m: [0;32mLinux[0m
I0216 17:48:18.585399 305464 kubeadm.go:322] [0;37mCGROUPS_CPU[0m: [0;32menabled[0m
I0216 17:48:18.585457 305464 kubeadm.go:322] [0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
I0216 17:48:18.585514 305464 kubeadm.go:322] [0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
I0216 17:48:18.585579 305464 kubeadm.go:322] [0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
I0216 17:48:18.585644 305464 kubeadm.go:322] [0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
I0216 17:48:18.585722 305464 kubeadm.go:322] [0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
I0216 17:48:18.585821 305464 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0216 17:48:18.585926 305464 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0216 17:48:18.586029 305464 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0216 17:48:18.586152 305464 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0216 17:48:18.586246 305464 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0216 17:48:18.586308 305464 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0216 17:48:18.586378 305464 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0216 17:48:18.588552 305464 out.go:204] - Generating certificates and keys ...
I0216 17:48:18.588664 305464 kubeadm.go:322] [certs] Using existing ca certificate authority
I0216 17:48:18.588733 305464 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0216 17:48:18.588801 305464 kubeadm.go:322] [certs] Generating "apiserver-kubelet-client" certificate and key
I0216 17:48:18.588858 305464 kubeadm.go:322] [certs] Generating "front-proxy-ca" certificate and key
I0216 17:48:18.588918 305464 kubeadm.go:322] [certs] Generating "front-proxy-client" certificate and key
I0216 17:48:18.588970 305464 kubeadm.go:322] [certs] Generating "etcd/ca" certificate and key
I0216 17:48:18.589024 305464 kubeadm.go:322] [certs] Generating "etcd/server" certificate and key
I0216 17:48:18.589145 305464 kubeadm.go:322] [certs] etcd/server serving cert is signed for DNS names [old-k8s-version-488384 localhost] and IPs [192.168.67.2 127.0.0.1 ::1]
I0216 17:48:18.589197 305464 kubeadm.go:322] [certs] Generating "etcd/peer" certificate and key
I0216 17:48:18.589314 305464 kubeadm.go:322] [certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-488384 localhost] and IPs [192.168.67.2 127.0.0.1 ::1]
I0216 17:48:18.589377 305464 kubeadm.go:322] [certs] Generating "etcd/healthcheck-client" certificate and key
I0216 17:48:18.589447 305464 kubeadm.go:322] [certs] Generating "apiserver-etcd-client" certificate and key
I0216 17:48:18.589494 305464 kubeadm.go:322] [certs] Generating "sa" key and public key
I0216 17:48:18.589553 305464 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0216 17:48:18.589604 305464 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0216 17:48:18.589657 305464 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0216 17:48:18.589720 305464 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0216 17:48:18.589773 305464 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0216 17:48:18.589838 305464 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0216 17:48:18.591974 305464 out.go:204] - Booting up control plane ...
I0216 17:48:18.592112 305464 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0216 17:48:18.592221 305464 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0216 17:48:18.592295 305464 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0216 17:48:18.592379 305464 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0216 17:48:18.592529 305464 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0216 17:48:18.592578 305464 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0216 17:48:18.592588 305464 kubeadm.go:322]
I0216 17:48:18.592625 305464 kubeadm.go:322] Unfortunately, an error has occurred:
I0216 17:48:18.592753 305464 kubeadm.go:322] timed out waiting for the condition
I0216 17:48:18.592766 305464 kubeadm.go:322]
I0216 17:48:18.592825 305464 kubeadm.go:322] This error is likely caused by:
I0216 17:48:18.592871 305464 kubeadm.go:322] - The kubelet is not running
I0216 17:48:18.592981 305464 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0216 17:48:18.592992 305464 kubeadm.go:322]
I0216 17:48:18.593089 305464 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0216 17:48:18.593130 305464 kubeadm.go:322] - 'systemctl status kubelet'
I0216 17:48:18.593170 305464 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0216 17:48:18.593181 305464 kubeadm.go:322]
I0216 17:48:18.593277 305464 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0216 17:48:18.593377 305464 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0216 17:48:18.593457 305464 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0216 17:48:18.593506 305464 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0216 17:48:18.593587 305464 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
W0216 17:48:18.593753 305464 out.go:239] ! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1053-aws[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-488384 localhost] and IPs [192.168.67.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-488384 localhost] and IPs [192.168.67.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1053-aws\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
! initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1053-aws[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [old-k8s-version-488384 localhost] and IPs [192.168.67.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [old-k8s-version-488384 localhost] and IPs [192.168.67.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1053-aws\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
I0216 17:48:18.593809 305464 ssh_runner.go:195] Run: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm reset --cri-socket /var/run/dockershim.sock --force"
I0216 17:48:18.594062 305464 kubeadm.go:322] - 'docker logs CONTAINERID'
I0216 17:48:19.406942 305464 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service kubelet
I0216 17:48:19.418708 305464 kubeadm.go:226] ignoring SystemVerification for kubeadm because of docker driver
I0216 17:48:19.418775 305464 ssh_runner.go:195] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0216 17:48:19.427933 305464 kubeadm.go:152] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:
stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0216 17:48:19.427977 305464 ssh_runner.go:286] Start: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0216 17:48:19.486298 305464 kubeadm.go:322] [init] Using Kubernetes version: v1.16.0
I0216 17:48:19.486637 305464 kubeadm.go:322] [preflight] Running pre-flight checks
I0216 17:48:19.684847 305464 kubeadm.go:322] [preflight] The system verification failed. Printing the output from the verification:
I0216 17:48:19.684927 305464 kubeadm.go:322] [0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1053-aws[0m
I0216 17:48:19.684979 305464 kubeadm.go:322] [0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
I0216 17:48:19.685017 305464 kubeadm.go:322] [0;37mOS[0m: [0;32mLinux[0m
I0216 17:48:19.685063 305464 kubeadm.go:322] [0;37mCGROUPS_CPU[0m: [0;32menabled[0m
I0216 17:48:19.685112 305464 kubeadm.go:322] [0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
I0216 17:48:19.685159 305464 kubeadm.go:322] [0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
I0216 17:48:19.685204 305464 kubeadm.go:322] [0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
I0216 17:48:19.685251 305464 kubeadm.go:322] [0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
I0216 17:48:19.685294 305464 kubeadm.go:322] [0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
I0216 17:48:19.779568 305464 kubeadm.go:322] [preflight] Pulling images required for setting up a Kubernetes cluster
I0216 17:48:19.779849 305464 kubeadm.go:322] [preflight] This might take a minute or two, depending on the speed of your internet connection
I0216 17:48:19.779964 305464 kubeadm.go:322] [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I0216 17:48:19.993242 305464 kubeadm.go:322] [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
I0216 17:48:19.995165 305464 kubeadm.go:322] [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
I0216 17:48:20.005297 305464 kubeadm.go:322] [kubelet-start] Activating the kubelet service
I0216 17:48:20.099817 305464 kubeadm.go:322] [certs] Using certificateDir folder "/var/lib/minikube/certs"
I0216 17:48:20.102229 305464 out.go:204] - Generating certificates and keys ...
I0216 17:48:20.102567 305464 kubeadm.go:322] [certs] Using existing ca certificate authority
I0216 17:48:20.103406 305464 kubeadm.go:322] [certs] Using existing apiserver certificate and key on disk
I0216 17:48:20.105503 305464 kubeadm.go:322] [certs] Using existing apiserver-kubelet-client certificate and key on disk
I0216 17:48:20.106142 305464 kubeadm.go:322] [certs] Using existing front-proxy-ca certificate authority
I0216 17:48:20.106381 305464 kubeadm.go:322] [certs] Using existing front-proxy-client certificate and key on disk
I0216 17:48:20.106697 305464 kubeadm.go:322] [certs] Using existing etcd/ca certificate authority
I0216 17:48:20.107285 305464 kubeadm.go:322] [certs] Using existing etcd/server certificate and key on disk
I0216 17:48:20.107871 305464 kubeadm.go:322] [certs] Using existing etcd/peer certificate and key on disk
I0216 17:48:20.108490 305464 kubeadm.go:322] [certs] Using existing etcd/healthcheck-client certificate and key on disk
I0216 17:48:20.109086 305464 kubeadm.go:322] [certs] Using existing apiserver-etcd-client certificate and key on disk
I0216 17:48:20.109282 305464 kubeadm.go:322] [certs] Using the existing "sa" key
I0216 17:48:20.109362 305464 kubeadm.go:322] [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I0216 17:48:20.370456 305464 kubeadm.go:322] [kubeconfig] Writing "admin.conf" kubeconfig file
I0216 17:48:20.660810 305464 kubeadm.go:322] [kubeconfig] Writing "kubelet.conf" kubeconfig file
I0216 17:48:21.120291 305464 kubeadm.go:322] [kubeconfig] Writing "controller-manager.conf" kubeconfig file
I0216 17:48:21.586099 305464 kubeadm.go:322] [kubeconfig] Writing "scheduler.conf" kubeconfig file
I0216 17:48:21.587413 305464 kubeadm.go:322] [control-plane] Using manifest folder "/etc/kubernetes/manifests"
I0216 17:48:21.589875 305464 out.go:204] - Booting up control plane ...
I0216 17:48:21.589973 305464 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-apiserver"
I0216 17:48:21.600186 305464 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-controller-manager"
I0216 17:48:21.601663 305464 kubeadm.go:322] [control-plane] Creating static Pod manifest for "kube-scheduler"
I0216 17:48:21.607908 305464 kubeadm.go:322] [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I0216 17:48:21.627176 305464 kubeadm.go:322] [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I0216 17:49:01.628166 305464 kubeadm.go:322] [kubelet-check] Initial timeout of 40s passed.
I0216 17:52:21.628720 305464 kubeadm.go:322]
I0216 17:52:21.628851 305464 kubeadm.go:322] Unfortunately, an error has occurred:
I0216 17:52:21.628938 305464 kubeadm.go:322] timed out waiting for the condition
I0216 17:52:21.628973 305464 kubeadm.go:322]
I0216 17:52:21.629038 305464 kubeadm.go:322] This error is likely caused by:
I0216 17:52:21.629088 305464 kubeadm.go:322] - The kubelet is not running
I0216 17:52:21.629225 305464 kubeadm.go:322] - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
I0216 17:52:21.629251 305464 kubeadm.go:322]
I0216 17:52:21.629386 305464 kubeadm.go:322] If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
I0216 17:52:21.629448 305464 kubeadm.go:322] - 'systemctl status kubelet'
I0216 17:52:21.629508 305464 kubeadm.go:322] - 'journalctl -xeu kubelet'
I0216 17:52:21.629533 305464 kubeadm.go:322]
I0216 17:52:21.629667 305464 kubeadm.go:322] Additionally, a control plane component may have crashed or exited when started by the container runtime.
I0216 17:52:21.629783 305464 kubeadm.go:322] To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
I0216 17:52:21.629931 305464 kubeadm.go:322] Here is one example how you may list all Kubernetes containers running in docker:
I0216 17:52:21.630032 305464 kubeadm.go:322] - 'docker ps -a | grep kube | grep -v pause'
I0216 17:52:21.630116 305464 kubeadm.go:322] Once you have found the failing container, you can inspect its logs with:
I0216 17:52:21.630149 305464 kubeadm.go:322] - 'docker logs CONTAINERID'
I0216 17:52:21.634731 305464 kubeadm.go:322] [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
I0216 17:52:21.634878 305464 kubeadm.go:322] [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
I0216 17:52:21.635092 305464 kubeadm.go:322] [WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1053-aws\n", err: exit status 1
I0216 17:52:21.635197 305464 kubeadm.go:322] [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
I0216 17:52:21.635282 305464 kubeadm.go:322] error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
I0216 17:52:21.635348 305464 kubeadm.go:322] To see the stack trace of this error execute with --v=5 or higher
I0216 17:52:21.635404 305464 kubeadm.go:406] StartCluster complete in 8m12.031297885s
I0216 17:52:21.635484 305464 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-apiserver --format={{.ID}}
I0216 17:52:21.657053 305464 logs.go:276] 0 containers: []
W0216 17:52:21.657115 305464 logs.go:278] No container was found matching "kube-apiserver"
I0216 17:52:21.657219 305464 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_etcd --format={{.ID}}
I0216 17:52:21.708381 305464 logs.go:276] 0 containers: []
W0216 17:52:21.708445 305464 logs.go:278] No container was found matching "etcd"
I0216 17:52:21.708519 305464 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_coredns --format={{.ID}}
I0216 17:52:21.750765 305464 logs.go:276] 0 containers: []
W0216 17:52:21.750832 305464 logs.go:278] No container was found matching "coredns"
I0216 17:52:21.750906 305464 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-scheduler --format={{.ID}}
I0216 17:52:21.771360 305464 logs.go:276] 0 containers: []
W0216 17:52:21.771416 305464 logs.go:278] No container was found matching "kube-scheduler"
I0216 17:52:21.771487 305464 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-proxy --format={{.ID}}
I0216 17:52:21.810962 305464 logs.go:276] 0 containers: []
W0216 17:52:21.811035 305464 logs.go:278] No container was found matching "kube-proxy"
I0216 17:52:21.811108 305464 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kube-controller-manager --format={{.ID}}
I0216 17:52:21.835509 305464 logs.go:276] 0 containers: []
W0216 17:52:21.835571 305464 logs.go:278] No container was found matching "kube-controller-manager"
I0216 17:52:21.835649 305464 ssh_runner.go:195] Run: docker ps -a --filter=name=k8s_kindnet --format={{.ID}}
I0216 17:52:21.870681 305464 logs.go:276] 0 containers: []
W0216 17:52:21.870747 305464 logs.go:278] No container was found matching "kindnet"
I0216 17:52:21.870771 305464 logs.go:123] Gathering logs for kubelet ...
I0216 17:52:21.870797 305464 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
W0216 17:52:21.906074 305464 logs.go:138] Found kubelet problem: Feb 16 17:52:01 old-k8s-version-488384 kubelet[5456]: E0216 17:52:01.519546 5456 pod_workers.go:191] Error syncing pod 9e914279041b7fec183513988f6a94cb ("kube-apiserver-old-k8s-version-488384_kube-system(9e914279041b7fec183513988f6a94cb)"), skipping: failed to "StartContainer" for "kube-apiserver" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-apiserver:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-apiserver:v1.16.0\" is not set"
W0216 17:52:21.918767 305464 logs.go:138] Found kubelet problem: Feb 16 17:52:06 old-k8s-version-488384 kubelet[5456]: E0216 17:52:06.513912 5456 pod_workers.go:191] Error syncing pod f5bdbfcedf2bccc429ba471ffe3804b7 ("etcd-old-k8s-version-488384_kube-system(f5bdbfcedf2bccc429ba471ffe3804b7)"), skipping: failed to "StartContainer" for "etcd" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/etcd:3.3.15-0\": Id or size of image \"k8s.gcr.io/etcd:3.3.15-0\" is not set"
W0216 17:52:21.921872 305464 logs.go:138] Found kubelet problem: Feb 16 17:52:07 old-k8s-version-488384 kubelet[5456]: E0216 17:52:07.514080 5456 pod_workers.go:191] Error syncing pod b3d303074fe0ca1d42a8bd9ed248df09 ("kube-scheduler-old-k8s-version-488384_kube-system(b3d303074fe0ca1d42a8bd9ed248df09)"), skipping: failed to "StartContainer" for "kube-scheduler" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-scheduler:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-scheduler:v1.16.0\" is not set"
W0216 17:52:21.927091 305464 logs.go:138] Found kubelet problem: Feb 16 17:52:09 old-k8s-version-488384 kubelet[5456]: E0216 17:52:09.513366 5456 pod_workers.go:191] Error syncing pod 9fc427d2e6746d2b3f18846f6f0fcafb ("kube-controller-manager-old-k8s-version-488384_kube-system(9fc427d2e6746d2b3f18846f6f0fcafb)"), skipping: failed to "StartContainer" for "kube-controller-manager" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-controller-manager:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-controller-manager:v1.16.0\" is not set"
W0216 17:52:21.940063 305464 logs.go:138] Found kubelet problem: Feb 16 17:52:14 old-k8s-version-488384 kubelet[5456]: E0216 17:52:14.519080 5456 pod_workers.go:191] Error syncing pod 9e914279041b7fec183513988f6a94cb ("kube-apiserver-old-k8s-version-488384_kube-system(9e914279041b7fec183513988f6a94cb)"), skipping: failed to "StartContainer" for "kube-apiserver" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-apiserver:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-apiserver:v1.16.0\" is not set"
W0216 17:52:21.947385 305464 logs.go:138] Found kubelet problem: Feb 16 17:52:17 old-k8s-version-488384 kubelet[5456]: E0216 17:52:17.519955 5456 pod_workers.go:191] Error syncing pod f5bdbfcedf2bccc429ba471ffe3804b7 ("etcd-old-k8s-version-488384_kube-system(f5bdbfcedf2bccc429ba471ffe3804b7)"), skipping: failed to "StartContainer" for "etcd" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/etcd:3.3.15-0\": Id or size of image \"k8s.gcr.io/etcd:3.3.15-0\" is not set"
W0216 17:52:21.957673 305464 logs.go:138] Found kubelet problem: Feb 16 17:52:21 old-k8s-version-488384 kubelet[5456]: E0216 17:52:21.520708 5456 pod_workers.go:191] Error syncing pod b3d303074fe0ca1d42a8bd9ed248df09 ("kube-scheduler-old-k8s-version-488384_kube-system(b3d303074fe0ca1d42a8bd9ed248df09)"), skipping: failed to "StartContainer" for "kube-scheduler" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-scheduler:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-scheduler:v1.16.0\" is not set"
I0216 17:52:21.959080 305464 logs.go:123] Gathering logs for dmesg ...
I0216 17:52:21.959128 305464 ssh_runner.go:195] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0216 17:52:21.989349 305464 logs.go:123] Gathering logs for describe nodes ...
I0216 17:52:21.989490 305464 ssh_runner.go:195] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0216 17:52:22.139986 305464 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.16.0/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?
** /stderr **
I0216 17:52:22.140045 305464 logs.go:123] Gathering logs for Docker ...
I0216 17:52:22.140065 305464 ssh_runner.go:195] Run: /bin/bash -c "sudo journalctl -u docker -u cri-docker -n 400"
I0216 17:52:22.162817 305464 logs.go:123] Gathering logs for container status ...
I0216 17:52:22.162848 305464 ssh_runner.go:195] Run: /bin/bash -c "sudo `which crictl || echo crictl` ps -a || sudo docker ps -a"
W0216 17:52:22.221718 305464 out.go:364] Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1053-aws[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1053-aws\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0216 17:52:22.221799 305464 out.go:239] *
*
W0216 17:52:22.221879 305464 out.go:239] X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1053-aws[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1053-aws\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1053-aws[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1053-aws\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0216 17:52:22.222182 305464 out.go:239] *
*
W0216 17:52:22.223218 305464 out.go:239] ╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
╭─────────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ * If the above advice does not help, please let us know: │
│ https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ * Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────╯
I0216 17:52:22.249058 305464 out.go:177] X Problems detected in kubelet:
I0216 17:52:22.268938 305464 out.go:177] Feb 16 17:52:01 old-k8s-version-488384 kubelet[5456]: E0216 17:52:01.519546 5456 pod_workers.go:191] Error syncing pod 9e914279041b7fec183513988f6a94cb ("kube-apiserver-old-k8s-version-488384_kube-system(9e914279041b7fec183513988f6a94cb)"), skipping: failed to "StartContainer" for "kube-apiserver" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-apiserver:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-apiserver:v1.16.0\" is not set"
I0216 17:52:22.282915 305464 out.go:177] Feb 16 17:52:06 old-k8s-version-488384 kubelet[5456]: E0216 17:52:06.513912 5456 pod_workers.go:191] Error syncing pod f5bdbfcedf2bccc429ba471ffe3804b7 ("etcd-old-k8s-version-488384_kube-system(f5bdbfcedf2bccc429ba471ffe3804b7)"), skipping: failed to "StartContainer" for "etcd" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/etcd:3.3.15-0\": Id or size of image \"k8s.gcr.io/etcd:3.3.15-0\" is not set"
I0216 17:52:22.298636 305464 out.go:177] Feb 16 17:52:07 old-k8s-version-488384 kubelet[5456]: E0216 17:52:07.514080 5456 pod_workers.go:191] Error syncing pod b3d303074fe0ca1d42a8bd9ed248df09 ("kube-scheduler-old-k8s-version-488384_kube-system(b3d303074fe0ca1d42a8bd9ed248df09)"), skipping: failed to "StartContainer" for "kube-scheduler" with ImageInspectError: "Failed to inspect image \"k8s.gcr.io/kube-scheduler:v1.16.0\": Id or size of image \"k8s.gcr.io/kube-scheduler:v1.16.0\" is not set"
I0216 17:52:22.313884 305464 out.go:177]
W0216 17:52:22.321031 305464 out.go:239] X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1053-aws[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1053-aws\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
X Exiting due to K8S_KUBELET_NOT_RUNNING: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.16.0:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,NumCPU,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
[0;37mKERNEL_VERSION[0m: [0;32m5.15.0-1053-aws[0m
[0;37mDOCKER_VERSION[0m: [0;32m25.0.3[0m
[0;37mOS[0m: [0;32mLinux[0m
[0;37mCGROUPS_CPU[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUACCT[0m: [0;32menabled[0m
[0;37mCGROUPS_CPUSET[0m: [0;32menabled[0m
[0;37mCGROUPS_DEVICES[0m: [0;32menabled[0m
[0;37mCGROUPS_FREEZER[0m: [0;32menabled[0m
[0;37mCGROUPS_MEMORY[0m: [0;32menabled[0m
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/var/lib/minikube/certs"
[certs] Using existing ca certificate authority
[certs] Using existing apiserver certificate and key on disk
[certs] Using existing apiserver-kubelet-client certificate and key on disk
[certs] Using existing front-proxy-ca certificate authority
[certs] Using existing front-proxy-client certificate and key on disk
[certs] Using existing etcd/ca certificate authority
[certs] Using existing etcd/server certificate and key on disk
[certs] Using existing etcd/peer certificate and key on disk
[certs] Using existing etcd/healthcheck-client certificate and key on disk
[certs] Using existing apiserver-etcd-client certificate and key on disk
[certs] Using the existing "sa" key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
Unfortunately, an error has occurred:
timed out waiting for the condition
This error is likely caused by:
- The kubelet is not running
- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
- 'systemctl status kubelet'
- 'journalctl -xeu kubelet'
Additionally, a control plane component may have crashed or exited when started by the container runtime.
To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
Here is one example how you may list all Kubernetes containers running in docker:
- 'docker ps -a | grep kube | grep -v pause'
Once you have found the failing container, you can inspect its logs with:
- 'docker logs CONTAINERID'
stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 25.0.3. Latest validated version: 18.09
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1053-aws\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher
W0216 17:52:22.321104 305464 out.go:239] * Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
* Suggestion: Check output of 'journalctl -xeu kubelet', try passing --extra-config=kubelet.cgroup-driver=systemd to minikube start
W0216 17:52:22.321124 305464 out.go:239] * Related issue: https://github.com/kubernetes/minikube/issues/4172
* Related issue: https://github.com/kubernetes/minikube/issues/4172
I0216 17:52:22.354669 305464 out.go:177]
** /stderr **
start_stop_delete_test.go:188: failed starting minikube -first start-. args "out/minikube-linux-arm64 start -p old-k8s-version-488384 --memory=2200 --alsologtostderr --wait=true --kvm-network=default --kvm-qemu-uri=qemu:///system --disable-driver-mounts --keep-context=false --driver=docker --container-runtime=docker --kubernetes-version=v1.16.0": exit status 109
helpers_test.go:222: -----------------------post-mortem--------------------------------
helpers_test.go:230: ======> post-mortem[TestStartStop/group/old-k8s-version/serial/FirstStart]: docker inspect <======
helpers_test.go:231: (dbg) Run: docker inspect old-k8s-version-488384
helpers_test.go:235: (dbg) docker inspect old-k8s-version-488384:
-- stdout --
[
{
"Id": "2ad7a05058fe2549cddf95871269dd919b02c087d6bc4ea6c1c641b232f4238d",
"Created": "2024-02-16T17:43:51.781636674Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 306435,
"ExitCode": 0,
"Error": "",
"StartedAt": "2024-02-16T17:43:52.321549666Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:83c2925cbfe44b87b5c0672f16807927d87d8625e89de4dc154c45daaaa04b5b",
"ResolvConfPath": "/var/lib/docker/containers/2ad7a05058fe2549cddf95871269dd919b02c087d6bc4ea6c1c641b232f4238d/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/2ad7a05058fe2549cddf95871269dd919b02c087d6bc4ea6c1c641b232f4238d/hostname",
"HostsPath": "/var/lib/docker/containers/2ad7a05058fe2549cddf95871269dd919b02c087d6bc4ea6c1c641b232f4238d/hosts",
"LogPath": "/var/lib/docker/containers/2ad7a05058fe2549cddf95871269dd919b02c087d6bc4ea6c1c641b232f4238d/2ad7a05058fe2549cddf95871269dd919b02c087d6bc4ea6c1c641b232f4238d-json.log",
"Name": "/old-k8s-version-488384",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "unconfined",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro",
"old-k8s-version-488384:/var"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "old-k8s-version-488384",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": ""
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"ConsoleSize": [
0,
0
],
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "host",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"Isolation": "",
"CpuShares": 0,
"Memory": 2306867200,
"NanoCpus": 2000000000,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": [],
"BlkioDeviceWriteBps": [],
"BlkioDeviceReadIOps": [],
"BlkioDeviceWriteIOps": [],
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"MemoryReservation": 0,
"MemorySwap": 4613734400,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": [],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/7434aa8b437605996aed483edf7d3b633bc7acee8630f474670417e40cd0e621-init/diff:/var/lib/docker/overlay2/946a7b4f2791bd4745aa26fd1fdd5eefb03c154f3c1fd517458d1937bbb85039/diff",
"MergedDir": "/var/lib/docker/overlay2/7434aa8b437605996aed483edf7d3b633bc7acee8630f474670417e40cd0e621/merged",
"UpperDir": "/var/lib/docker/overlay2/7434aa8b437605996aed483edf7d3b633bc7acee8630f474670417e40cd0e621/diff",
"WorkDir": "/var/lib/docker/overlay2/7434aa8b437605996aed483edf7d3b633bc7acee8630f474670417e40cd0e621/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "old-k8s-version-488384",
"Source": "/var/lib/docker/volumes/old-k8s-version-488384/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "old-k8s-version-488384",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"22/tcp": {},
"2376/tcp": {},
"32443/tcp": {},
"5000/tcp": {},
"8443/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1708008208-17936@sha256:4ea1136332ba1476cda33a97bf12e2f96995cc120674fbafd3ade22d1118ecdf",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "old-k8s-version-488384",
"name.minikube.sigs.k8s.io": "old-k8s-version-488384",
"role.minikube.sigs.k8s.io": ""
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "1e99aebf69c0f66ee8799a92fbb9d9b35c73420d9fe016ee29fa9d199d23cdde",
"SandboxKey": "/var/run/docker/netns/1e99aebf69c0",
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33057"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33056"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33053"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33055"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "33054"
}
]
},
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"old-k8s-version-488384": {
"IPAMConfig": {
"IPv4Address": "192.168.67.2"
},
"Links": null,
"Aliases": [
"2ad7a05058fe",
"old-k8s-version-488384"
],
"MacAddress": "02:42:c0:a8:43:02",
"NetworkID": "066a8ce33ebb9a8bf9130346706c7668acc42f9f2a9352243a5b99995ed10eb4",
"EndpointID": "7f089d26c6ab52b2462e3bb9dd4eddece50cdc6dbfc4abbff695c3e2d07b6874",
"Gateway": "192.168.67.1",
"IPAddress": "192.168.67.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"DriverOpts": null,
"DNSNames": [
"old-k8s-version-488384",
"2ad7a05058fe"
]
}
}
}
}
]
-- /stdout --
helpers_test.go:239: (dbg) Run: out/minikube-linux-arm64 status --format={{.Host}} -p old-k8s-version-488384 -n old-k8s-version-488384
helpers_test.go:239: (dbg) Non-zero exit: out/minikube-linux-arm64 status --format={{.Host}} -p old-k8s-version-488384 -n old-k8s-version-488384: exit status 6 (352.007409ms)
-- stdout --
Running
WARNING: Your kubectl is pointing to stale minikube-vm.
To fix the kubectl context, run `minikube update-context`
-- /stdout --
** stderr **
E0216 17:52:22.821493 335872 status.go:415] kubeconfig endpoint: extract IP: "old-k8s-version-488384" does not appear in /home/jenkins/minikube-integration/17936-2208/kubeconfig
** /stderr **
helpers_test.go:239: status error: exit status 6 (may be ok)
helpers_test.go:241: "old-k8s-version-488384" host is not running, skipping log retrieval (state="Running\nWARNING: Your kubectl is pointing to stale minikube-vm.\nTo fix the kubectl context, run `minikube update-context`")
--- FAIL: TestStartStop/group/old-k8s-version/serial/FirstStart (517.00s)