=== RUN TestMultiControlPlane/serial/DegradedAfterClusterRestart
ha_test.go:390: (dbg) Run: out/minikube-linux-amd64 profile list --output json
ha_test.go:413: expected profile "ha-866665" in json of 'profile list' to have "Degraded" status but have "Stopped" status. got *"{\"invalid\":[],\"valid\":[{\"Name\":\"ha-866665\",\"Status\":\"Stopped\",\"Config\":{\"Name\":\"ha-866665\",\"KeepContext\":false,\"EmbedCerts\":false,\"MinikubeISO\":\"https://storage.googleapis.com/minikube-builds/iso/18213/minikube-v1.32.1-1710459732-18213-amd64.iso\",\"KicBaseImage\":\"gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1710284843-18375@sha256:d67c38c9fc2ad14c48d95e17cbac49314325db5758d8f7b3de60b927e62ce94f\",\"Memory\":2200,\"CPUs\":2,\"DiskSize\":20000,\"Driver\":\"kvm2\",\"HyperkitVpnKitSock\":\"\",\"HyperkitVSockPorts\":[],\"DockerEnv\":null,\"ContainerVolumeMounts\":null,\"InsecureRegistry\":null,\"RegistryMirror\":[],\"HostOnlyCIDR\":\"192.168.59.1/24\",\"HypervVirtualSwitch\":\"\",\"HypervUseExternalSwitch\":false,\"HypervExternalAdapter\":\"\",\"KVMNetwork\":\"default\",\"KVMQemuURI\":\"qemu:///system\",\"KVMGPU\":false,\"KVMHidden\":false,\"KVMNUMACount\":
1,\"APIServerPort\":8443,\"DockerOpt\":null,\"DisableDriverMounts\":false,\"NFSShare\":[],\"NFSSharesRoot\":\"/nfsshares\",\"UUID\":\"\",\"NoVTXCheck\":false,\"DNSProxy\":false,\"HostDNSResolver\":true,\"HostOnlyNicType\":\"virtio\",\"NatNicType\":\"virtio\",\"SSHIPAddress\":\"\",\"SSHUser\":\"root\",\"SSHKey\":\"\",\"SSHPort\":22,\"KubernetesConfig\":{\"KubernetesVersion\":\"v1.28.4\",\"ClusterName\":\"ha-866665\",\"Namespace\":\"default\",\"APIServerHAVIP\":\"192.168.39.254\",\"APIServerName\":\"minikubeCA\",\"APIServerNames\":null,\"APIServerIPs\":null,\"DNSDomain\":\"cluster.local\",\"ContainerRuntime\":\"crio\",\"CRISocket\":\"\",\"NetworkPlugin\":\"cni\",\"FeatureGates\":\"\",\"ServiceCIDR\":\"10.96.0.0/12\",\"ImageRepository\":\"\",\"LoadBalancerStartIP\":\"\",\"LoadBalancerEndIP\":\"\",\"CustomIngressCert\":\"\",\"RegistryAliases\":\"\",\"ExtraOptions\":null,\"ShouldLoadCachedImages\":true,\"EnableDefaultCNI\":false,\"CNI\":\"\"},\"Nodes\":[{\"Name\":\"\",\"IP\":\"192.168.39.78\",\"Port\":8443,\"Kuber
netesVersion\":\"v1.28.4\",\"ContainerRuntime\":\"crio\",\"ControlPlane\":true,\"Worker\":true},{\"Name\":\"m02\",\"IP\":\"192.168.39.27\",\"Port\":8443,\"KubernetesVersion\":\"v1.28.4\",\"ContainerRuntime\":\"crio\",\"ControlPlane\":true,\"Worker\":true},{\"Name\":\"m04\",\"IP\":\"192.168.39.184\",\"Port\":0,\"KubernetesVersion\":\"v1.28.4\",\"ContainerRuntime\":\"crio\",\"ControlPlane\":false,\"Worker\":true}],\"Addons\":{\"ambassador\":false,\"auto-pause\":false,\"cloud-spanner\":false,\"csi-hostpath-driver\":false,\"dashboard\":false,\"default-storageclass\":false,\"efk\":false,\"freshpod\":false,\"gcp-auth\":false,\"gvisor\":false,\"headlamp\":false,\"helm-tiller\":false,\"inaccel\":false,\"ingress\":false,\"ingress-dns\":false,\"inspektor-gadget\":false,\"istio\":false,\"istio-provisioner\":false,\"kong\":false,\"kubeflow\":false,\"kubevirt\":false,\"logviewer\":false,\"metallb\":false,\"metrics-server\":false,\"nvidia-device-plugin\":false,\"nvidia-driver-installer\":false,\"nvidia-gpu-device-plugin\":
false,\"olm\":false,\"pod-security-policy\":false,\"portainer\":false,\"registry\":false,\"registry-aliases\":false,\"registry-creds\":false,\"storage-provisioner\":false,\"storage-provisioner-gluster\":false,\"storage-provisioner-rancher\":false,\"volumesnapshots\":false,\"yakd\":false},\"CustomAddonImages\":null,\"CustomAddonRegistries\":null,\"VerifyComponents\":{\"apiserver\":true,\"apps_running\":true,\"default_sa\":true,\"extra\":true,\"kubelet\":true,\"node_ready\":true,\"system_pods\":true},\"StartHostTimeout\":360000000000,\"ScheduledStop\":null,\"ExposedPorts\":[],\"ListenAddress\":\"\",\"Network\":\"\",\"Subnet\":\"\",\"MultiNodeRequested\":true,\"ExtraDisks\":0,\"CertExpiration\":94608000000000000,\"Mount\":false,\"MountString\":\"/home/jenkins:/minikube-host\",\"Mount9PVersion\":\"9p2000.L\",\"MountGID\":\"docker\",\"MountIP\":\"\",\"MountMSize\":262144,\"MountOptions\":[],\"MountPort\":0,\"MountType\":\"9p\",\"MountUID\":\"docker\",\"BinaryMirror\":\"\",\"DisableOptimizations\":false,\"DisableMe
trics\":false,\"CustomQemuFirmwarePath\":\"\",\"SocketVMnetClientPath\":\"\",\"SocketVMnetPath\":\"\",\"StaticIP\":\"\",\"SSHAuthSock\":\"\",\"SSHAgentPID\":0,\"GPUs\":\"\",\"AutoPauseInterval\":60000000000},\"Active\":false}]}"*. args: "out/minikube-linux-amd64 profile list --output json"
helpers_test.go:222: -----------------------post-mortem--------------------------------
helpers_test.go:239: (dbg) Run: out/minikube-linux-amd64 status --format={{.Host}} -p ha-866665 -n ha-866665
helpers_test.go:239: (dbg) Non-zero exit: out/minikube-linux-amd64 status --format={{.Host}} -p ha-866665 -n ha-866665: exit status 2 (232.68542ms)
-- stdout --
Running
-- /stdout --
helpers_test.go:239: status error: exit status 2 (may be ok)
helpers_test.go:244: <<< TestMultiControlPlane/serial/DegradedAfterClusterRestart FAILED: start of post-mortem logs <<<
helpers_test.go:245: ======> post-mortem[TestMultiControlPlane/serial/DegradedAfterClusterRestart]: minikube logs <======
helpers_test.go:247: (dbg) Run: out/minikube-linux-amd64 -p ha-866665 logs -n 25
helpers_test.go:247: (dbg) Done: out/minikube-linux-amd64 -p ha-866665 logs -n 25: (1.417671079s)
helpers_test.go:252: TestMultiControlPlane/serial/DegradedAfterClusterRestart logs:
-- stdout --
==> Audit <==
|---------|----------------------------------------------------------------------------------|-----------|---------|---------|---------------------|---------------------|
| Command | Args | Profile | User | Version | Start Time | End Time |
|---------|----------------------------------------------------------------------------------|-----------|---------|---------|---------------------|---------------------|
| cp | ha-866665 cp ha-866665-m03:/home/docker/cp-test.txt | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | ha-866665-m04:/home/docker/cp-test_ha-866665-m03_ha-866665-m04.txt | | | | | |
| ssh | ha-866665 ssh -n | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | ha-866665-m03 sudo cat | | | | | |
| | /home/docker/cp-test.txt | | | | | |
| ssh | ha-866665 ssh -n ha-866665-m04 sudo cat | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | /home/docker/cp-test_ha-866665-m03_ha-866665-m04.txt | | | | | |
| cp | ha-866665 cp testdata/cp-test.txt | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | ha-866665-m04:/home/docker/cp-test.txt | | | | | |
| ssh | ha-866665 ssh -n | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | ha-866665-m04 sudo cat | | | | | |
| | /home/docker/cp-test.txt | | | | | |
| cp | ha-866665 cp ha-866665-m04:/home/docker/cp-test.txt | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | /tmp/TestMultiControlPlaneserialCopyFile3271648541/001/cp-test_ha-866665-m04.txt | | | | | |
| ssh | ha-866665 ssh -n | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | ha-866665-m04 sudo cat | | | | | |
| | /home/docker/cp-test.txt | | | | | |
| cp | ha-866665 cp ha-866665-m04:/home/docker/cp-test.txt | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | ha-866665:/home/docker/cp-test_ha-866665-m04_ha-866665.txt | | | | | |
| ssh | ha-866665 ssh -n | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | ha-866665-m04 sudo cat | | | | | |
| | /home/docker/cp-test.txt | | | | | |
| ssh | ha-866665 ssh -n ha-866665 sudo cat | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | /home/docker/cp-test_ha-866665-m04_ha-866665.txt | | | | | |
| cp | ha-866665 cp ha-866665-m04:/home/docker/cp-test.txt | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | ha-866665-m02:/home/docker/cp-test_ha-866665-m04_ha-866665-m02.txt | | | | | |
| ssh | ha-866665 ssh -n | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | ha-866665-m04 sudo cat | | | | | |
| | /home/docker/cp-test.txt | | | | | |
| ssh | ha-866665 ssh -n ha-866665-m02 sudo cat | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | /home/docker/cp-test_ha-866665-m04_ha-866665-m02.txt | | | | | |
| cp | ha-866665 cp ha-866665-m04:/home/docker/cp-test.txt | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | ha-866665-m03:/home/docker/cp-test_ha-866665-m04_ha-866665-m03.txt | | | | | |
| ssh | ha-866665 ssh -n | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | ha-866665-m04 sudo cat | | | | | |
| | /home/docker/cp-test.txt | | | | | |
| ssh | ha-866665 ssh -n ha-866665-m03 sudo cat | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | 15 Mar 24 06:15 UTC |
| | /home/docker/cp-test_ha-866665-m04_ha-866665-m03.txt | | | | | |
| node | ha-866665 node stop m02 -v=7 | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:15 UTC | |
| | --alsologtostderr | | | | | |
| node | ha-866665 node start m02 -v=7 | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:17 UTC | |
| | --alsologtostderr | | | | | |
| node | list -p ha-866665 -v=7 | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:18 UTC | |
| | --alsologtostderr | | | | | |
| stop | -p ha-866665 -v=7 | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:18 UTC | |
| | --alsologtostderr | | | | | |
| start | -p ha-866665 --wait=true -v=7 | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:20 UTC | |
| | --alsologtostderr | | | | | |
| node | list -p ha-866665 | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:24 UTC | |
| node | ha-866665 node delete m03 -v=7 | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:24 UTC | 15 Mar 24 06:25 UTC |
| | --alsologtostderr | | | | | |
| stop | ha-866665 stop -v=7 | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:25 UTC | |
| | --alsologtostderr | | | | | |
| start | -p ha-866665 --wait=true | ha-866665 | jenkins | v1.32.0 | 15 Mar 24 06:28 UTC | |
| | -v=7 --alsologtostderr | | | | | |
| | --driver=kvm2 | | | | | |
| | --container-runtime=crio | | | | | |
|---------|----------------------------------------------------------------------------------|-----------|---------|---------|---------------------|---------------------|
==> Last Start <==
Log file created at: 2024/03/15 06:28:00
Running on machine: ubuntu-20-agent-8
Binary: Built with gc go1.22.1 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0315 06:28:00.069231 33437 out.go:291] Setting OutFile to fd 1 ...
I0315 06:28:00.069368 33437 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0315 06:28:00.069377 33437 out.go:304] Setting ErrFile to fd 2...
I0315 06:28:00.069382 33437 out.go:338] TERM=,COLORTERM=, which probably does not support color
I0315 06:28:00.069568 33437 root.go:338] Updating PATH: /home/jenkins/minikube-integration/18213-8825/.minikube/bin
I0315 06:28:00.070093 33437 out.go:298] Setting JSON to false
I0315 06:28:00.070988 33437 start.go:129] hostinfo: {"hostname":"ubuntu-20-agent-8","uptime":4176,"bootTime":1710479904,"procs":193,"os":"linux","platform":"ubuntu","platformFamily":"debian","platformVersion":"20.04","kernelVersion":"5.15.0-1053-gcp","kernelArch":"x86_64","virtualizationSystem":"kvm","virtualizationRole":"guest","hostId":"591c9f12-2938-3743-e2bf-c56a050d43d1"}
I0315 06:28:00.071057 33437 start.go:139] virtualization: kvm guest
I0315 06:28:00.074620 33437 out.go:177] * [ha-866665] minikube v1.32.0 on Ubuntu 20.04 (kvm/amd64)
I0315 06:28:00.076308 33437 out.go:177] - MINIKUBE_LOCATION=18213
I0315 06:28:00.076319 33437 notify.go:220] Checking for updates...
I0315 06:28:00.079197 33437 out.go:177] - MINIKUBE_SUPPRESS_DOCKER_PERFORMANCE=true
I0315 06:28:00.080588 33437 out.go:177] - KUBECONFIG=/home/jenkins/minikube-integration/18213-8825/kubeconfig
I0315 06:28:00.081864 33437 out.go:177] - MINIKUBE_HOME=/home/jenkins/minikube-integration/18213-8825/.minikube
I0315 06:28:00.083324 33437 out.go:177] - MINIKUBE_BIN=out/minikube-linux-amd64
I0315 06:28:00.084651 33437 out.go:177] - MINIKUBE_FORCE_SYSTEMD=
I0315 06:28:00.086650 33437 config.go:182] Loaded profile config "ha-866665": Driver=kvm2, ContainerRuntime=crio, KubernetesVersion=v1.28.4
I0315 06:28:00.087036 33437 main.go:141] libmachine: Found binary path at /home/jenkins/workspace/KVM_Linux_crio_integration/out/docker-machine-driver-kvm2
I0315 06:28:00.087091 33437 main.go:141] libmachine: Launching plugin server for driver kvm2
I0315 06:28:00.102114 33437 main.go:141] libmachine: Plugin server listening at address 127.0.0.1:36355
I0315 06:28:00.102558 33437 main.go:141] libmachine: () Calling .GetVersion
I0315 06:28:00.103095 33437 main.go:141] libmachine: Using API Version 1
I0315 06:28:00.103124 33437 main.go:141] libmachine: () Calling .SetConfigRaw
I0315 06:28:00.103438 33437 main.go:141] libmachine: () Calling .GetMachineName
I0315 06:28:00.103601 33437 main.go:141] libmachine: (ha-866665) Calling .DriverName
I0315 06:28:00.103876 33437 driver.go:392] Setting default libvirt URI to qemu:///system
I0315 06:28:00.104159 33437 main.go:141] libmachine: Found binary path at /home/jenkins/workspace/KVM_Linux_crio_integration/out/docker-machine-driver-kvm2
I0315 06:28:00.104210 33437 main.go:141] libmachine: Launching plugin server for driver kvm2
I0315 06:28:00.119133 33437 main.go:141] libmachine: Plugin server listening at address 127.0.0.1:35375
I0315 06:28:00.119585 33437 main.go:141] libmachine: () Calling .GetVersion
I0315 06:28:00.120070 33437 main.go:141] libmachine: Using API Version 1
I0315 06:28:00.120090 33437 main.go:141] libmachine: () Calling .SetConfigRaw
I0315 06:28:00.120437 33437 main.go:141] libmachine: () Calling .GetMachineName
I0315 06:28:00.120651 33437 main.go:141] libmachine: (ha-866665) Calling .DriverName
I0315 06:28:00.156291 33437 out.go:177] * Using the kvm2 driver based on existing profile
I0315 06:28:00.157886 33437 start.go:297] selected driver: kvm2
I0315 06:28:00.157902 33437 start.go:901] validating driver "kvm2" against &{Name:ha-866665 KeepContext:false EmbedCerts:false MinikubeISO:https://storage.googleapis.com/minikube-builds/iso/18213/minikube-v1.32.1-1710459732-18213-amd64.iso KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1710284843-18375@sha256:d67c38c9fc2ad14c48d95e17cbac49314325db5758d8f7b3de60b927e62ce94f Memory:2200 CPUs:2 DiskSize:20000 Driver:kvm2 HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVer
sion:v1.28.4 ClusterName:ha-866665 Namespace:default APIServerHAVIP:192.168.39.254 APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:crio CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.39.78 Port:8443 KubernetesVersion:v1.28.4 ContainerRuntime:crio ControlPlane:true Worker:true} {Name:m02 IP:192.168.39.27 Port:8443 KubernetesVersion:v1.28.4 ContainerRuntime:crio ControlPlane:true Worker:true} {Name:m04 IP:192.168.39.184 Port:0 KubernetesVersion:v1.28.4 ContainerRuntime: ControlPlane:false Worker:true}] Addons:map[ambassador:false auto-pause:false cloud-spanner:false csi-hostpath-driver:false dashboard:false default-storageclass:false efk:false freshpod:false gcp-auth:false gvisor:false headlamp:false helm-tiller:false inaccel:false ingress:false ingres
s-dns:false inspektor-gadget:false istio:false istio-provisioner:false kong:false kubeflow:false kubevirt:false logviewer:false metallb:false metrics-server:false nvidia-device-plugin:false nvidia-driver-installer:false nvidia-gpu-device-plugin:false olm:false pod-security-policy:false portainer:false registry:false registry-aliases:false registry-creds:false storage-provisioner:false storage-provisioner-gluster:false storage-provisioner-rancher:false volumesnapshots:false yakd:false] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:true ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOpt
imizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0315 06:28:00.158040 33437 start.go:912] status for kvm2: {Installed:true Healthy:true Running:true NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0315 06:28:00.158357 33437 install.go:52] acquiring lock: {Name:mk900956b073697a4aa6c80a27c6bb0742a99a53 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0315 06:28:00.158422 33437 install.go:117] Validating docker-machine-driver-kvm2, PATH=/home/jenkins/minikube-integration/18213-8825/.minikube/bin:/home/jenkins/workspace/KVM_Linux_crio_integration/out/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/local/go/bin:/home/jenkins/go/bin:/usr/local/bin/:/usr/local/go/bin/:/home/jenkins/go/bin
I0315 06:28:00.174458 33437 install.go:137] /home/jenkins/workspace/KVM_Linux_crio_integration/out/docker-machine-driver-kvm2 version is 1.32.0
I0315 06:28:00.175133 33437 start_flags.go:947] Waiting for all components: map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true]
I0315 06:28:00.175191 33437 cni.go:84] Creating CNI manager for ""
I0315 06:28:00.175203 33437 cni.go:136] multinode detected (3 nodes found), recommending kindnet
I0315 06:28:00.175251 33437 start.go:340] cluster config:
{Name:ha-866665 KeepContext:false EmbedCerts:false MinikubeISO:https://storage.googleapis.com/minikube-builds/iso/18213/minikube-v1.32.1-1710459732-18213-amd64.iso KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1710284843-18375@sha256:d67c38c9fc2ad14c48d95e17cbac49314325db5758d8f7b3de60b927e62ce94f Memory:2200 CPUs:2 DiskSize:20000 Driver:kvm2 HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.28.4 ClusterName:ha-866665 Namespace:default APIServerHAVIP:192.168.39
.254 APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:crio CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.39.78 Port:8443 KubernetesVersion:v1.28.4 ContainerRuntime:crio ControlPlane:true Worker:true} {Name:m02 IP:192.168.39.27 Port:8443 KubernetesVersion:v1.28.4 ContainerRuntime:crio ControlPlane:true Worker:true} {Name:m04 IP:192.168.39.184 Port:0 KubernetesVersion:v1.28.4 ContainerRuntime:crio ControlPlane:false Worker:true}] Addons:map[ambassador:false auto-pause:false cloud-spanner:false csi-hostpath-driver:false dashboard:false default-storageclass:false efk:false freshpod:false gcp-auth:false gvisor:false headlamp:false helm-tiller:false inaccel:false ingress:false ingress-dns:false inspektor-gadget:false istio:false istio-provisioner:false kon
g:false kubeflow:false kubevirt:false logviewer:false metallb:false metrics-server:false nvidia-device-plugin:false nvidia-driver-installer:false nvidia-gpu-device-plugin:false olm:false pod-security-policy:false portainer:false registry:false registry-aliases:false registry-creds:false storage-provisioner:false storage-provisioner-gluster:false storage-provisioner-rancher:false volumesnapshots:false yakd:false] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:true ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetC
lientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0315 06:28:00.175362 33437 iso.go:125] acquiring lock: {Name:mk048c52a0305dbd9220f756a753e61a2b267b56 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0315 06:28:00.177468 33437 out.go:177] * Starting "ha-866665" primary control-plane node in "ha-866665" cluster
I0315 06:28:00.179008 33437 preload.go:132] Checking if preload exists for k8s version v1.28.4 and runtime crio
I0315 06:28:00.179040 33437 preload.go:147] Found local preload: /home/jenkins/minikube-integration/18213-8825/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.28.4-cri-o-overlay-amd64.tar.lz4
I0315 06:28:00.179047 33437 cache.go:56] Caching tarball of preloaded images
I0315 06:28:00.179131 33437 preload.go:173] Found /home/jenkins/minikube-integration/18213-8825/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.28.4-cri-o-overlay-amd64.tar.lz4 in cache, skipping download
I0315 06:28:00.179142 33437 cache.go:59] Finished verifying existence of preloaded tar for v1.28.4 on crio
I0315 06:28:00.179294 33437 profile.go:142] Saving config to /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/config.json ...
I0315 06:28:00.179480 33437 start.go:360] acquireMachinesLock for ha-866665: {Name:mk586b481ecda7c454c5ea28c054514c88c0788e Clock:{} Delay:500ms Timeout:13m0s Cancel:<nil>}
I0315 06:28:00.179520 33437 start.go:364] duration metric: took 23.255µs to acquireMachinesLock for "ha-866665"
I0315 06:28:00.179534 33437 start.go:96] Skipping create...Using existing machine configuration
I0315 06:28:00.179545 33437 fix.go:54] fixHost starting:
I0315 06:28:00.179780 33437 main.go:141] libmachine: Found binary path at /home/jenkins/workspace/KVM_Linux_crio_integration/out/docker-machine-driver-kvm2
I0315 06:28:00.179810 33437 main.go:141] libmachine: Launching plugin server for driver kvm2
I0315 06:28:00.194943 33437 main.go:141] libmachine: Plugin server listening at address 127.0.0.1:35043
I0315 06:28:00.195338 33437 main.go:141] libmachine: () Calling .GetVersion
I0315 06:28:00.195810 33437 main.go:141] libmachine: Using API Version 1
I0315 06:28:00.195828 33437 main.go:141] libmachine: () Calling .SetConfigRaw
I0315 06:28:00.196117 33437 main.go:141] libmachine: () Calling .GetMachineName
I0315 06:28:00.196309 33437 main.go:141] libmachine: (ha-866665) Calling .DriverName
I0315 06:28:00.196495 33437 main.go:141] libmachine: (ha-866665) Calling .GetState
I0315 06:28:00.198137 33437 fix.go:112] recreateIfNeeded on ha-866665: state=Running err=<nil>
W0315 06:28:00.198153 33437 fix.go:138] unexpected machine state, will restart: <nil>
I0315 06:28:00.200161 33437 out.go:177] * Updating the running kvm2 "ha-866665" VM ...
I0315 06:28:00.201473 33437 machine.go:94] provisionDockerMachine start ...
I0315 06:28:00.201503 33437 main.go:141] libmachine: (ha-866665) Calling .DriverName
I0315 06:28:00.201694 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHHostname
I0315 06:28:00.204348 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:28:00.204777 33437 main.go:141] libmachine: (ha-866665) DBG | found host DHCP lease matching {name: "", mac: "52:54:00:96:55:9d", ip: ""} in network mk-ha-866665: {Iface:virbr1 ExpiryTime:2024-03-15 07:10:36 +0000 UTC Type:0 Mac:52:54:00:96:55:9d Iaid: IPaddr:192.168.39.78 Prefix:24 Hostname:ha-866665 Clientid:01:52:54:00:96:55:9d}
I0315 06:28:00.204797 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined IP address 192.168.39.78 and MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:28:00.204937 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHPort
I0315 06:28:00.205101 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHKeyPath
I0315 06:28:00.205264 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHKeyPath
I0315 06:28:00.205376 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHUsername
I0315 06:28:00.205519 33437 main.go:141] libmachine: Using SSH client type: native
I0315 06:28:00.205700 33437 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x82d4a0] 0x830200 <nil> [] 0s} 192.168.39.78 22 <nil> <nil>}
I0315 06:28:00.205711 33437 main.go:141] libmachine: About to run SSH command:
hostname
I0315 06:28:00.305507 33437 main.go:141] libmachine: SSH cmd err, output: <nil>: ha-866665
I0315 06:28:00.305537 33437 main.go:141] libmachine: (ha-866665) Calling .GetMachineName
I0315 06:28:00.305774 33437 buildroot.go:166] provisioning hostname "ha-866665"
I0315 06:28:00.305803 33437 main.go:141] libmachine: (ha-866665) Calling .GetMachineName
I0315 06:28:00.305989 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHHostname
I0315 06:28:00.308802 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:28:00.309169 33437 main.go:141] libmachine: (ha-866665) DBG | found host DHCP lease matching {name: "", mac: "52:54:00:96:55:9d", ip: ""} in network mk-ha-866665: {Iface:virbr1 ExpiryTime:2024-03-15 07:10:36 +0000 UTC Type:0 Mac:52:54:00:96:55:9d Iaid: IPaddr:192.168.39.78 Prefix:24 Hostname:ha-866665 Clientid:01:52:54:00:96:55:9d}
I0315 06:28:00.309190 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined IP address 192.168.39.78 and MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:28:00.309354 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHPort
I0315 06:28:00.309553 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHKeyPath
I0315 06:28:00.309826 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHKeyPath
I0315 06:28:00.310014 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHUsername
I0315 06:28:00.310190 33437 main.go:141] libmachine: Using SSH client type: native
I0315 06:28:00.310366 33437 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x82d4a0] 0x830200 <nil> [] 0s} 192.168.39.78 22 <nil> <nil>}
I0315 06:28:00.310382 33437 main.go:141] libmachine: About to run SSH command:
sudo hostname ha-866665 && echo "ha-866665" | sudo tee /etc/hostname
I0315 06:28:00.429403 33437 main.go:141] libmachine: SSH cmd err, output: <nil>: ha-866665
I0315 06:28:00.429432 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHHostname
I0315 06:28:00.432235 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:28:00.432606 33437 main.go:141] libmachine: (ha-866665) DBG | found host DHCP lease matching {name: "", mac: "52:54:00:96:55:9d", ip: ""} in network mk-ha-866665: {Iface:virbr1 ExpiryTime:2024-03-15 07:10:36 +0000 UTC Type:0 Mac:52:54:00:96:55:9d Iaid: IPaddr:192.168.39.78 Prefix:24 Hostname:ha-866665 Clientid:01:52:54:00:96:55:9d}
I0315 06:28:00.432644 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined IP address 192.168.39.78 and MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:28:00.432809 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHPort
I0315 06:28:00.432999 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHKeyPath
I0315 06:28:00.433159 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHKeyPath
I0315 06:28:00.433289 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHUsername
I0315 06:28:00.433507 33437 main.go:141] libmachine: Using SSH client type: native
I0315 06:28:00.433711 33437 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x82d4a0] 0x830200 <nil> [] 0s} 192.168.39.78 22 <nil> <nil>}
I0315 06:28:00.433736 33437 main.go:141] libmachine: About to run SSH command:
if ! grep -xq '.*\sha-866665' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 ha-866665/g' /etc/hosts;
else
echo '127.0.1.1 ha-866665' | sudo tee -a /etc/hosts;
fi
fi
I0315 06:28:00.533992 33437 main.go:141] libmachine: SSH cmd err, output: <nil>:
I0315 06:28:00.534024 33437 buildroot.go:172] set auth options {CertDir:/home/jenkins/minikube-integration/18213-8825/.minikube CaCertPath:/home/jenkins/minikube-integration/18213-8825/.minikube/certs/ca.pem CaPrivateKeyPath:/home/jenkins/minikube-integration/18213-8825/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/home/jenkins/minikube-integration/18213-8825/.minikube/machines/server.pem ServerKeyPath:/home/jenkins/minikube-integration/18213-8825/.minikube/machines/server-key.pem ClientKeyPath:/home/jenkins/minikube-integration/18213-8825/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/home/jenkins/minikube-integration/18213-8825/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/home/jenkins/minikube-integration/18213-8825/.minikube}
I0315 06:28:00.534042 33437 buildroot.go:174] setting up certificates
I0315 06:28:00.534050 33437 provision.go:84] configureAuth start
I0315 06:28:00.534059 33437 main.go:141] libmachine: (ha-866665) Calling .GetMachineName
I0315 06:28:00.534324 33437 main.go:141] libmachine: (ha-866665) Calling .GetIP
I0315 06:28:00.536932 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:28:00.537280 33437 main.go:141] libmachine: (ha-866665) DBG | found host DHCP lease matching {name: "", mac: "52:54:00:96:55:9d", ip: ""} in network mk-ha-866665: {Iface:virbr1 ExpiryTime:2024-03-15 07:10:36 +0000 UTC Type:0 Mac:52:54:00:96:55:9d Iaid: IPaddr:192.168.39.78 Prefix:24 Hostname:ha-866665 Clientid:01:52:54:00:96:55:9d}
I0315 06:28:00.537309 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined IP address 192.168.39.78 and MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:28:00.537403 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHHostname
I0315 06:28:00.539778 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:28:00.540170 33437 main.go:141] libmachine: (ha-866665) DBG | found host DHCP lease matching {name: "", mac: "52:54:00:96:55:9d", ip: ""} in network mk-ha-866665: {Iface:virbr1 ExpiryTime:2024-03-15 07:10:36 +0000 UTC Type:0 Mac:52:54:00:96:55:9d Iaid: IPaddr:192.168.39.78 Prefix:24 Hostname:ha-866665 Clientid:01:52:54:00:96:55:9d}
I0315 06:28:00.540188 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined IP address 192.168.39.78 and MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:28:00.540352 33437 provision.go:143] copyHostCerts
I0315 06:28:00.540374 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/certs/ca.pem -> /home/jenkins/minikube-integration/18213-8825/.minikube/ca.pem
I0315 06:28:00.540409 33437 exec_runner.go:144] found /home/jenkins/minikube-integration/18213-8825/.minikube/ca.pem, removing ...
I0315 06:28:00.540418 33437 exec_runner.go:203] rm: /home/jenkins/minikube-integration/18213-8825/.minikube/ca.pem
I0315 06:28:00.540502 33437 exec_runner.go:151] cp: /home/jenkins/minikube-integration/18213-8825/.minikube/certs/ca.pem --> /home/jenkins/minikube-integration/18213-8825/.minikube/ca.pem (1078 bytes)
I0315 06:28:00.540577 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/certs/cert.pem -> /home/jenkins/minikube-integration/18213-8825/.minikube/cert.pem
I0315 06:28:00.540595 33437 exec_runner.go:144] found /home/jenkins/minikube-integration/18213-8825/.minikube/cert.pem, removing ...
I0315 06:28:00.540602 33437 exec_runner.go:203] rm: /home/jenkins/minikube-integration/18213-8825/.minikube/cert.pem
I0315 06:28:00.540626 33437 exec_runner.go:151] cp: /home/jenkins/minikube-integration/18213-8825/.minikube/certs/cert.pem --> /home/jenkins/minikube-integration/18213-8825/.minikube/cert.pem (1123 bytes)
I0315 06:28:00.540689 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/certs/key.pem -> /home/jenkins/minikube-integration/18213-8825/.minikube/key.pem
I0315 06:28:00.540712 33437 exec_runner.go:144] found /home/jenkins/minikube-integration/18213-8825/.minikube/key.pem, removing ...
I0315 06:28:00.540721 33437 exec_runner.go:203] rm: /home/jenkins/minikube-integration/18213-8825/.minikube/key.pem
I0315 06:28:00.540757 33437 exec_runner.go:151] cp: /home/jenkins/minikube-integration/18213-8825/.minikube/certs/key.pem --> /home/jenkins/minikube-integration/18213-8825/.minikube/key.pem (1679 bytes)
I0315 06:28:00.540858 33437 provision.go:117] generating server cert: /home/jenkins/minikube-integration/18213-8825/.minikube/machines/server.pem ca-key=/home/jenkins/minikube-integration/18213-8825/.minikube/certs/ca.pem private-key=/home/jenkins/minikube-integration/18213-8825/.minikube/certs/ca-key.pem org=jenkins.ha-866665 san=[127.0.0.1 192.168.39.78 ha-866665 localhost minikube]
I0315 06:28:00.727324 33437 provision.go:177] copyRemoteCerts
I0315 06:28:00.727392 33437 ssh_runner.go:195] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0315 06:28:00.727415 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHHostname
I0315 06:28:00.730386 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:28:00.730795 33437 main.go:141] libmachine: (ha-866665) DBG | found host DHCP lease matching {name: "", mac: "52:54:00:96:55:9d", ip: ""} in network mk-ha-866665: {Iface:virbr1 ExpiryTime:2024-03-15 07:10:36 +0000 UTC Type:0 Mac:52:54:00:96:55:9d Iaid: IPaddr:192.168.39.78 Prefix:24 Hostname:ha-866665 Clientid:01:52:54:00:96:55:9d}
I0315 06:28:00.730817 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined IP address 192.168.39.78 and MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:28:00.731033 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHPort
I0315 06:28:00.731269 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHKeyPath
I0315 06:28:00.731448 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHUsername
I0315 06:28:00.731603 33437 sshutil.go:53] new ssh client: &{IP:192.168.39.78 Port:22 SSHKeyPath:/home/jenkins/minikube-integration/18213-8825/.minikube/machines/ha-866665/id_rsa Username:docker}
I0315 06:28:00.811679 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/machines/server.pem -> /etc/docker/server.pem
I0315 06:28:00.811760 33437 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18213-8825/.minikube/machines/server.pem --> /etc/docker/server.pem (1196 bytes)
I0315 06:28:00.840244 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/machines/server-key.pem -> /etc/docker/server-key.pem
I0315 06:28:00.840325 33437 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18213-8825/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1675 bytes)
I0315 06:28:00.866687 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/certs/ca.pem -> /etc/docker/ca.pem
I0315 06:28:00.866766 33437 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18213-8825/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1078 bytes)
I0315 06:28:00.893745 33437 provision.go:87] duration metric: took 359.681699ms to configureAuth
I0315 06:28:00.893783 33437 buildroot.go:189] setting minikube options for container-runtime
I0315 06:28:00.894043 33437 config.go:182] Loaded profile config "ha-866665": Driver=kvm2, ContainerRuntime=crio, KubernetesVersion=v1.28.4
I0315 06:28:00.894134 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHHostname
I0315 06:28:00.897023 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:28:00.897388 33437 main.go:141] libmachine: (ha-866665) DBG | found host DHCP lease matching {name: "", mac: "52:54:00:96:55:9d", ip: ""} in network mk-ha-866665: {Iface:virbr1 ExpiryTime:2024-03-15 07:10:36 +0000 UTC Type:0 Mac:52:54:00:96:55:9d Iaid: IPaddr:192.168.39.78 Prefix:24 Hostname:ha-866665 Clientid:01:52:54:00:96:55:9d}
I0315 06:28:00.897411 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined IP address 192.168.39.78 and MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:28:00.897569 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHPort
I0315 06:28:00.897752 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHKeyPath
I0315 06:28:00.897920 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHKeyPath
I0315 06:28:00.898052 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHUsername
I0315 06:28:00.898189 33437 main.go:141] libmachine: Using SSH client type: native
I0315 06:28:00.898433 33437 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x82d4a0] 0x830200 <nil> [] 0s} 192.168.39.78 22 <nil> <nil>}
I0315 06:28:00.898471 33437 main.go:141] libmachine: About to run SSH command:
sudo mkdir -p /etc/sysconfig && printf %!s(MISSING) "
CRIO_MINIKUBE_OPTIONS='--insecure-registry 10.96.0.0/12 '
" | sudo tee /etc/sysconfig/crio.minikube && sudo systemctl restart crio
I0315 06:29:35.718292 33437 main.go:141] libmachine: SSH cmd err, output: <nil>:
CRIO_MINIKUBE_OPTIONS='--insecure-registry 10.96.0.0/12 '
I0315 06:29:35.718325 33437 machine.go:97] duration metric: took 1m35.516837024s to provisionDockerMachine
I0315 06:29:35.718343 33437 start.go:293] postStartSetup for "ha-866665" (driver="kvm2")
I0315 06:29:35.718359 33437 start.go:322] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0315 06:29:35.718374 33437 main.go:141] libmachine: (ha-866665) Calling .DriverName
I0315 06:29:35.718720 33437 ssh_runner.go:195] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0315 06:29:35.718757 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHHostname
I0315 06:29:35.722200 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:29:35.722789 33437 main.go:141] libmachine: (ha-866665) DBG | found host DHCP lease matching {name: "", mac: "52:54:00:96:55:9d", ip: ""} in network mk-ha-866665: {Iface:virbr1 ExpiryTime:2024-03-15 07:10:36 +0000 UTC Type:0 Mac:52:54:00:96:55:9d Iaid: IPaddr:192.168.39.78 Prefix:24 Hostname:ha-866665 Clientid:01:52:54:00:96:55:9d}
I0315 06:29:35.722838 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined IP address 192.168.39.78 and MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:29:35.722915 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHPort
I0315 06:29:35.723113 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHKeyPath
I0315 06:29:35.723278 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHUsername
I0315 06:29:35.723452 33437 sshutil.go:53] new ssh client: &{IP:192.168.39.78 Port:22 SSHKeyPath:/home/jenkins/minikube-integration/18213-8825/.minikube/machines/ha-866665/id_rsa Username:docker}
I0315 06:29:35.808948 33437 ssh_runner.go:195] Run: cat /etc/os-release
I0315 06:29:35.813922 33437 info.go:137] Remote host: Buildroot 2023.02.9
I0315 06:29:35.813958 33437 filesync.go:126] Scanning /home/jenkins/minikube-integration/18213-8825/.minikube/addons for local assets ...
I0315 06:29:35.814035 33437 filesync.go:126] Scanning /home/jenkins/minikube-integration/18213-8825/.minikube/files for local assets ...
I0315 06:29:35.814150 33437 filesync.go:149] local asset: /home/jenkins/minikube-integration/18213-8825/.minikube/files/etc/ssl/certs/160752.pem -> 160752.pem in /etc/ssl/certs
I0315 06:29:35.814165 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/files/etc/ssl/certs/160752.pem -> /etc/ssl/certs/160752.pem
I0315 06:29:35.814262 33437 ssh_runner.go:195] Run: sudo mkdir -p /etc/ssl/certs
I0315 06:29:35.825162 33437 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18213-8825/.minikube/files/etc/ssl/certs/160752.pem --> /etc/ssl/certs/160752.pem (1708 bytes)
I0315 06:29:35.853607 33437 start.go:296] duration metric: took 135.248885ms for postStartSetup
I0315 06:29:35.853656 33437 main.go:141] libmachine: (ha-866665) Calling .DriverName
I0315 06:29:35.853968 33437 ssh_runner.go:195] Run: sudo ls --almost-all -1 /var/lib/minikube/backup
I0315 06:29:35.853999 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHHostname
I0315 06:29:35.857046 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:29:35.857515 33437 main.go:141] libmachine: (ha-866665) DBG | found host DHCP lease matching {name: "", mac: "52:54:00:96:55:9d", ip: ""} in network mk-ha-866665: {Iface:virbr1 ExpiryTime:2024-03-15 07:10:36 +0000 UTC Type:0 Mac:52:54:00:96:55:9d Iaid: IPaddr:192.168.39.78 Prefix:24 Hostname:ha-866665 Clientid:01:52:54:00:96:55:9d}
I0315 06:29:35.857538 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined IP address 192.168.39.78 and MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:29:35.857740 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHPort
I0315 06:29:35.857904 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHKeyPath
I0315 06:29:35.858174 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHUsername
I0315 06:29:35.858327 33437 sshutil.go:53] new ssh client: &{IP:192.168.39.78 Port:22 SSHKeyPath:/home/jenkins/minikube-integration/18213-8825/.minikube/machines/ha-866665/id_rsa Username:docker}
W0315 06:29:35.939552 33437 fix.go:99] cannot read backup folder, skipping restore: read dir: sudo ls --almost-all -1 /var/lib/minikube/backup: Process exited with status 2
stdout:
stderr:
ls: cannot access '/var/lib/minikube/backup': No such file or directory
I0315 06:29:35.939581 33437 fix.go:56] duration metric: took 1m35.76003955s for fixHost
I0315 06:29:35.939603 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHHostname
I0315 06:29:35.942284 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:29:35.942621 33437 main.go:141] libmachine: (ha-866665) DBG | found host DHCP lease matching {name: "", mac: "52:54:00:96:55:9d", ip: ""} in network mk-ha-866665: {Iface:virbr1 ExpiryTime:2024-03-15 07:10:36 +0000 UTC Type:0 Mac:52:54:00:96:55:9d Iaid: IPaddr:192.168.39.78 Prefix:24 Hostname:ha-866665 Clientid:01:52:54:00:96:55:9d}
I0315 06:29:35.942656 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined IP address 192.168.39.78 and MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:29:35.942842 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHPort
I0315 06:29:35.943040 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHKeyPath
I0315 06:29:35.943209 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHKeyPath
I0315 06:29:35.943341 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHUsername
I0315 06:29:35.943527 33437 main.go:141] libmachine: Using SSH client type: native
I0315 06:29:35.943686 33437 main.go:141] libmachine: &{{{<nil> 0 [] [] []} docker [0x82d4a0] 0x830200 <nil> [] 0s} 192.168.39.78 22 <nil> <nil>}
I0315 06:29:35.943696 33437 main.go:141] libmachine: About to run SSH command:
date +%!s(MISSING).%!N(MISSING)
I0315 06:29:36.045713 33437 main.go:141] libmachine: SSH cmd err, output: <nil>: 1710484176.008370872
I0315 06:29:36.045741 33437 fix.go:216] guest clock: 1710484176.008370872
I0315 06:29:36.045749 33437 fix.go:229] Guest: 2024-03-15 06:29:36.008370872 +0000 UTC Remote: 2024-03-15 06:29:35.939588087 +0000 UTC m=+95.917046644 (delta=68.782785ms)
I0315 06:29:36.045784 33437 fix.go:200] guest clock delta is within tolerance: 68.782785ms
I0315 06:29:36.045790 33437 start.go:83] releasing machines lock for "ha-866665", held for 1m35.866260772s
I0315 06:29:36.045808 33437 main.go:141] libmachine: (ha-866665) Calling .DriverName
I0315 06:29:36.046095 33437 main.go:141] libmachine: (ha-866665) Calling .GetIP
I0315 06:29:36.048748 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:29:36.049090 33437 main.go:141] libmachine: (ha-866665) DBG | found host DHCP lease matching {name: "", mac: "52:54:00:96:55:9d", ip: ""} in network mk-ha-866665: {Iface:virbr1 ExpiryTime:2024-03-15 07:10:36 +0000 UTC Type:0 Mac:52:54:00:96:55:9d Iaid: IPaddr:192.168.39.78 Prefix:24 Hostname:ha-866665 Clientid:01:52:54:00:96:55:9d}
I0315 06:29:36.049125 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined IP address 192.168.39.78 and MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:29:36.049284 33437 main.go:141] libmachine: (ha-866665) Calling .DriverName
I0315 06:29:36.049937 33437 main.go:141] libmachine: (ha-866665) Calling .DriverName
I0315 06:29:36.050138 33437 main.go:141] libmachine: (ha-866665) Calling .DriverName
I0315 06:29:36.050191 33437 ssh_runner.go:195] Run: curl -sS -m 2 https://registry.k8s.io/
I0315 06:29:36.050244 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHHostname
I0315 06:29:36.050342 33437 ssh_runner.go:195] Run: cat /version.json
I0315 06:29:36.050361 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHHostname
I0315 06:29:36.053057 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:29:36.053439 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:29:36.053473 33437 main.go:141] libmachine: (ha-866665) DBG | found host DHCP lease matching {name: "", mac: "52:54:00:96:55:9d", ip: ""} in network mk-ha-866665: {Iface:virbr1 ExpiryTime:2024-03-15 07:10:36 +0000 UTC Type:0 Mac:52:54:00:96:55:9d Iaid: IPaddr:192.168.39.78 Prefix:24 Hostname:ha-866665 Clientid:01:52:54:00:96:55:9d}
I0315 06:29:36.053529 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined IP address 192.168.39.78 and MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:29:36.053632 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHPort
I0315 06:29:36.053795 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHKeyPath
I0315 06:29:36.053957 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHUsername
I0315 06:29:36.053958 33437 main.go:141] libmachine: (ha-866665) DBG | found host DHCP lease matching {name: "", mac: "52:54:00:96:55:9d", ip: ""} in network mk-ha-866665: {Iface:virbr1 ExpiryTime:2024-03-15 07:10:36 +0000 UTC Type:0 Mac:52:54:00:96:55:9d Iaid: IPaddr:192.168.39.78 Prefix:24 Hostname:ha-866665 Clientid:01:52:54:00:96:55:9d}
I0315 06:29:36.053983 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined IP address 192.168.39.78 and MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:29:36.054117 33437 sshutil.go:53] new ssh client: &{IP:192.168.39.78 Port:22 SSHKeyPath:/home/jenkins/minikube-integration/18213-8825/.minikube/machines/ha-866665/id_rsa Username:docker}
I0315 06:29:36.054145 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHPort
I0315 06:29:36.054330 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHKeyPath
I0315 06:29:36.054470 33437 main.go:141] libmachine: (ha-866665) Calling .GetSSHUsername
I0315 06:29:36.054647 33437 sshutil.go:53] new ssh client: &{IP:192.168.39.78 Port:22 SSHKeyPath:/home/jenkins/minikube-integration/18213-8825/.minikube/machines/ha-866665/id_rsa Username:docker}
I0315 06:29:36.174816 33437 ssh_runner.go:195] Run: systemctl --version
I0315 06:29:36.181715 33437 ssh_runner.go:195] Run: sudo sh -c "podman version >/dev/null"
I0315 06:29:36.358096 33437 ssh_runner.go:195] Run: sh -c "stat /etc/cni/net.d/*loopback.conf*"
W0315 06:29:36.367581 33437 cni.go:209] loopback cni configuration skipped: "/etc/cni/net.d/*loopback.conf*" not found
I0315 06:29:36.367659 33437 ssh_runner.go:195] Run: sudo find /etc/cni/net.d -maxdepth 1 -type f ( ( -name *bridge* -or -name *podman* ) -and -not -name *.mk_disabled ) -printf "%!p(MISSING), " -exec sh -c "sudo mv {} {}.mk_disabled" ;
I0315 06:29:36.383454 33437 cni.go:259] no active bridge cni configs found in "/etc/cni/net.d" - nothing to disable
I0315 06:29:36.383480 33437 start.go:494] detecting cgroup driver to use...
I0315 06:29:36.383550 33437 ssh_runner.go:195] Run: sudo systemctl stop -f containerd
I0315 06:29:36.407514 33437 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service containerd
I0315 06:29:36.425757 33437 docker.go:217] disabling cri-docker service (if available) ...
I0315 06:29:36.425807 33437 ssh_runner.go:195] Run: sudo systemctl stop -f cri-docker.socket
I0315 06:29:36.448161 33437 ssh_runner.go:195] Run: sudo systemctl stop -f cri-docker.service
I0315 06:29:36.466873 33437 ssh_runner.go:195] Run: sudo systemctl disable cri-docker.socket
I0315 06:29:36.634934 33437 ssh_runner.go:195] Run: sudo systemctl mask cri-docker.service
I0315 06:29:36.809139 33437 docker.go:233] disabling docker service ...
I0315 06:29:36.809211 33437 ssh_runner.go:195] Run: sudo systemctl stop -f docker.socket
I0315 06:29:36.831715 33437 ssh_runner.go:195] Run: sudo systemctl stop -f docker.service
I0315 06:29:36.847966 33437 ssh_runner.go:195] Run: sudo systemctl disable docker.socket
I0315 06:29:37.006211 33437 ssh_runner.go:195] Run: sudo systemctl mask docker.service
I0315 06:29:37.162186 33437 ssh_runner.go:195] Run: sudo systemctl is-active --quiet service docker
I0315 06:29:37.178537 33437 ssh_runner.go:195] Run: /bin/bash -c "sudo mkdir -p /etc && printf %!s(MISSING) "runtime-endpoint: unix:///var/run/crio/crio.sock
" | sudo tee /etc/crictl.yaml"
I0315 06:29:37.200300 33437 crio.go:59] configure cri-o to use "registry.k8s.io/pause:3.9" pause image...
I0315 06:29:37.200368 33437 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|^.*pause_image = .*$|pause_image = "registry.k8s.io/pause:3.9"|' /etc/crio/crio.conf.d/02-crio.conf"
I0315 06:29:37.212398 33437 crio.go:70] configuring cri-o to use "cgroupfs" as cgroup driver...
I0315 06:29:37.212455 33437 ssh_runner.go:195] Run: sh -c "sudo sed -i 's|^.*cgroup_manager = .*$|cgroup_manager = "cgroupfs"|' /etc/crio/crio.conf.d/02-crio.conf"
I0315 06:29:37.223908 33437 ssh_runner.go:195] Run: sh -c "sudo sed -i '/conmon_cgroup = .*/d' /etc/crio/crio.conf.d/02-crio.conf"
I0315 06:29:37.235824 33437 ssh_runner.go:195] Run: sh -c "sudo sed -i '/cgroup_manager = .*/a conmon_cgroup = "pod"' /etc/crio/crio.conf.d/02-crio.conf"
I0315 06:29:37.247520 33437 ssh_runner.go:195] Run: sh -c "sudo rm -rf /etc/cni/net.mk"
I0315 06:29:37.259008 33437 ssh_runner.go:195] Run: sudo sysctl net.bridge.bridge-nf-call-iptables
I0315 06:29:37.269062 33437 ssh_runner.go:195] Run: sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward"
I0315 06:29:37.281152 33437 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0315 06:29:37.434941 33437 ssh_runner.go:195] Run: sudo systemctl restart crio
I0315 06:31:11.689384 33437 ssh_runner.go:235] Completed: sudo systemctl restart crio: (1m34.254391387s)
I0315 06:31:11.689430 33437 start.go:541] Will wait 60s for socket path /var/run/crio/crio.sock
I0315 06:31:11.689496 33437 ssh_runner.go:195] Run: stat /var/run/crio/crio.sock
I0315 06:31:11.698102 33437 start.go:562] Will wait 60s for crictl version
I0315 06:31:11.698154 33437 ssh_runner.go:195] Run: which crictl
I0315 06:31:11.702605 33437 ssh_runner.go:195] Run: sudo /usr/bin/crictl version
I0315 06:31:11.746302 33437 start.go:578] Version: 0.1.0
RuntimeName: cri-o
RuntimeVersion: 1.29.1
RuntimeApiVersion: v1
I0315 06:31:11.746373 33437 ssh_runner.go:195] Run: crio --version
I0315 06:31:11.777004 33437 ssh_runner.go:195] Run: crio --version
I0315 06:31:11.813410 33437 out.go:177] * Preparing Kubernetes v1.28.4 on CRI-O 1.29.1 ...
I0315 06:31:11.815123 33437 main.go:141] libmachine: (ha-866665) Calling .GetIP
I0315 06:31:11.818257 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:31:11.818696 33437 main.go:141] libmachine: (ha-866665) DBG | found host DHCP lease matching {name: "", mac: "52:54:00:96:55:9d", ip: ""} in network mk-ha-866665: {Iface:virbr1 ExpiryTime:2024-03-15 07:10:36 +0000 UTC Type:0 Mac:52:54:00:96:55:9d Iaid: IPaddr:192.168.39.78 Prefix:24 Hostname:ha-866665 Clientid:01:52:54:00:96:55:9d}
I0315 06:31:11.818717 33437 main.go:141] libmachine: (ha-866665) DBG | domain ha-866665 has defined IP address 192.168.39.78 and MAC address 52:54:00:96:55:9d in network mk-ha-866665
I0315 06:31:11.818982 33437 ssh_runner.go:195] Run: grep 192.168.39.1 host.minikube.internal$ /etc/hosts
I0315 06:31:11.824253 33437 kubeadm.go:877] updating cluster {Name:ha-866665 KeepContext:false EmbedCerts:false MinikubeISO:https://storage.googleapis.com/minikube-builds/iso/18213/minikube-v1.32.1-1710459732-18213-amd64.iso KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1710284843-18375@sha256:d67c38c9fc2ad14c48d95e17cbac49314325db5758d8f7b3de60b927e62ce94f Memory:2200 CPUs:2 DiskSize:20000 Driver:kvm2 HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.28.4 Cl
usterName:ha-866665 Namespace:default APIServerHAVIP:192.168.39.254 APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:crio CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.39.78 Port:8443 KubernetesVersion:v1.28.4 ContainerRuntime:crio ControlPlane:true Worker:true} {Name:m02 IP:192.168.39.27 Port:8443 KubernetesVersion:v1.28.4 ContainerRuntime:crio ControlPlane:true Worker:true} {Name:m04 IP:192.168.39.184 Port:0 KubernetesVersion:v1.28.4 ContainerRuntime:crio ControlPlane:false Worker:true}] Addons:map[ambassador:false auto-pause:false cloud-spanner:false csi-hostpath-driver:false dashboard:false default-storageclass:false efk:false freshpod:false gcp-auth:false gvisor:false headlamp:false helm-tiller:false inaccel:false ingress:false ingress-dns:false
inspektor-gadget:false istio:false istio-provisioner:false kong:false kubeflow:false kubevirt:false logviewer:false metallb:false metrics-server:false nvidia-device-plugin:false nvidia-driver-installer:false nvidia-gpu-device-plugin:false olm:false pod-security-policy:false portainer:false registry:false registry-aliases:false registry-creds:false storage-provisioner:false storage-provisioner-gluster:false storage-provisioner-rancher:false volumesnapshots:false yakd:false] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:true ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:
false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s} ...
I0315 06:31:11.824379 33437 preload.go:132] Checking if preload exists for k8s version v1.28.4 and runtime crio
I0315 06:31:11.824419 33437 ssh_runner.go:195] Run: sudo crictl images --output json
I0315 06:31:11.877400 33437 crio.go:496] all images are preloaded for cri-o runtime.
I0315 06:31:11.877423 33437 crio.go:415] Images already preloaded, skipping extraction
I0315 06:31:11.877466 33437 ssh_runner.go:195] Run: sudo crictl images --output json
I0315 06:31:11.913358 33437 crio.go:496] all images are preloaded for cri-o runtime.
I0315 06:31:11.913383 33437 cache_images.go:84] Images are preloaded, skipping loading
I0315 06:31:11.913393 33437 kubeadm.go:928] updating node { 192.168.39.78 8443 v1.28.4 crio true true} ...
I0315 06:31:11.913524 33437 kubeadm.go:940] kubelet [Unit]
Wants=crio.service
[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.28.4/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --hostname-override=ha-866665 --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.39.78
[Install]
config:
{KubernetesVersion:v1.28.4 ClusterName:ha-866665 Namespace:default APIServerHAVIP:192.168.39.254 APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:crio CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:}
I0315 06:31:11.913604 33437 ssh_runner.go:195] Run: crio config
I0315 06:31:11.961648 33437 cni.go:84] Creating CNI manager for ""
I0315 06:31:11.961666 33437 cni.go:136] multinode detected (3 nodes found), recommending kindnet
I0315 06:31:11.961674 33437 kubeadm.go:84] Using pod CIDR: 10.244.0.0/16
I0315 06:31:11.961692 33437 kubeadm.go:181] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.39.78 APIServerPort:8443 KubernetesVersion:v1.28.4 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:ha-866665 NodeName:ha-866665 DNSDomain:cluster.local CRISocket:/var/run/crio/crio.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.39.78"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NodeIP:192.168.39.78 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticPodPath:/etc/kubernetes/m
anifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[] ResolvConfSearchRegression:false KubeletConfigOpts:map[containerRuntimeEndpoint:unix:///var/run/crio/crio.sock hairpinMode:hairpin-veth runtimeRequestTimeout:15m] PrependCriSocketUnix:true}
I0315 06:31:11.961854 33437 kubeadm.go:187] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.39.78
bindPort: 8443
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
ttl: 24h0m0s
usages:
- signing
- authentication
nodeRegistration:
criSocket: unix:///var/run/crio/crio.sock
name: "ha-866665"
kubeletExtraArgs:
node-ip: 192.168.39.78
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.39.78"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: mk
controlPlaneEndpoint: control-plane.minikube.internal:8443
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
proxy-refresh-interval: "70000"
kubernetesVersion: v1.28.4
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
containerRuntimeEndpoint: unix:///var/run/crio/crio.sock
hairpinMode: hairpin-veth
runtimeRequestTimeout: 15m
clusterDomain: "cluster.local"
# disable disk resource management by default
imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%!"(MISSING)
nodefs.inodesFree: "0%!"(MISSING)
imagefs.available: "0%!"(MISSING)
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout: 0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout: 0s
I0315 06:31:11.961877 33437 kube-vip.go:111] generating kube-vip config ...
I0315 06:31:11.961925 33437 ssh_runner.go:195] Run: sudo sh -c "modprobe --all ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack"
I0315 06:31:11.974783 33437 kube-vip.go:163] auto-enabling control-plane load-balancing in kube-vip
I0315 06:31:11.974879 33437 kube-vip.go:133] kube-vip config:
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: null
name: kube-vip
namespace: kube-system
spec:
containers:
- args:
- manager
env:
- name: vip_arp
value: "true"
- name: port
value: "8443"
- name: vip_interface
value: eth0
- name: vip_cidr
value: "32"
- name: dns_mode
value: first
- name: cp_enable
value: "true"
- name: cp_namespace
value: kube-system
- name: vip_leaderelection
value: "true"
- name: vip_leasename
value: plndr-cp-lock
- name: vip_leaseduration
value: "5"
- name: vip_renewdeadline
value: "3"
- name: vip_retryperiod
value: "1"
- name: address
value: 192.168.39.254
- name: prometheus_server
value: :2112
- name : lb_enable
value: "true"
- name: lb_port
value: "8443"
image: ghcr.io/kube-vip/kube-vip:v0.7.1
imagePullPolicy: IfNotPresent
name: kube-vip
resources: {}
securityContext:
capabilities:
add:
- NET_ADMIN
- NET_RAW
volumeMounts:
- mountPath: /etc/kubernetes/admin.conf
name: kubeconfig
hostAliases:
- hostnames:
- kubernetes
ip: 127.0.0.1
hostNetwork: true
volumes:
- hostPath:
path: "/etc/kubernetes/admin.conf"
name: kubeconfig
status: {}
I0315 06:31:11.974928 33437 ssh_runner.go:195] Run: sudo ls /var/lib/minikube/binaries/v1.28.4
I0315 06:31:11.985708 33437 binaries.go:44] Found k8s binaries, skipping transfer
I0315 06:31:11.985779 33437 ssh_runner.go:195] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube /etc/kubernetes/manifests
I0315 06:31:11.996849 33437 ssh_runner.go:362] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (308 bytes)
I0315 06:31:12.015133 33437 ssh_runner.go:362] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0315 06:31:12.032728 33437 ssh_runner.go:362] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (2150 bytes)
I0315 06:31:12.050473 33437 ssh_runner.go:362] scp memory --> /etc/kubernetes/manifests/kube-vip.yaml (1346 bytes)
I0315 06:31:12.071279 33437 ssh_runner.go:195] Run: grep 192.168.39.254 control-plane.minikube.internal$ /etc/hosts
I0315 06:31:12.075748 33437 ssh_runner.go:195] Run: sudo systemctl daemon-reload
I0315 06:31:12.239653 33437 ssh_runner.go:195] Run: sudo systemctl start kubelet
I0315 06:31:12.288563 33437 certs.go:68] Setting up /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665 for IP: 192.168.39.78
I0315 06:31:12.288592 33437 certs.go:194] generating shared ca certs ...
I0315 06:31:12.288612 33437 certs.go:226] acquiring lock for ca certs: {Name:mk21107528582bd739e76442bcf4baaf7000ed8d Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0315 06:31:12.288830 33437 certs.go:235] skipping valid "minikubeCA" ca cert: /home/jenkins/minikube-integration/18213-8825/.minikube/ca.key
I0315 06:31:12.288895 33437 certs.go:235] skipping valid "proxyClientCA" ca cert: /home/jenkins/minikube-integration/18213-8825/.minikube/proxy-client-ca.key
I0315 06:31:12.288911 33437 certs.go:256] generating profile certs ...
I0315 06:31:12.289016 33437 certs.go:359] skipping valid signed profile cert regeneration for "minikube-user": /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/client.key
I0315 06:31:12.289054 33437 certs.go:363] generating signed profile cert for "minikube": /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/apiserver.key.39db9211
I0315 06:31:12.289075 33437 crypto.go:68] Generating cert /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/apiserver.crt.39db9211 with IP's: [10.96.0.1 127.0.0.1 10.0.0.1 192.168.39.78 192.168.39.27 192.168.39.254]
I0315 06:31:12.459406 33437 crypto.go:156] Writing cert to /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/apiserver.crt.39db9211 ...
I0315 06:31:12.459436 33437 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/apiserver.crt.39db9211: {Name:mkce2140c17c76a43eac310ec6de314aee20f623 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0315 06:31:12.459623 33437 crypto.go:164] Writing key to /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/apiserver.key.39db9211 ...
I0315 06:31:12.459635 33437 lock.go:35] WriteFile acquiring /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/apiserver.key.39db9211: {Name:mk03c3e33d6e2b84dc52dfa74e4afefa164f8f54 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0315 06:31:12.459705 33437 certs.go:381] copying /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/apiserver.crt.39db9211 -> /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/apiserver.crt
I0315 06:31:12.459841 33437 certs.go:385] copying /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/apiserver.key.39db9211 -> /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/apiserver.key
I0315 06:31:12.459958 33437 certs.go:359] skipping valid signed profile cert regeneration for "aggregator": /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/proxy-client.key
I0315 06:31:12.459972 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/ca.crt -> /var/lib/minikube/certs/ca.crt
I0315 06:31:12.459985 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/ca.key -> /var/lib/minikube/certs/ca.key
I0315 06:31:12.459995 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/proxy-client-ca.crt -> /var/lib/minikube/certs/proxy-client-ca.crt
I0315 06:31:12.460005 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/proxy-client-ca.key -> /var/lib/minikube/certs/proxy-client-ca.key
I0315 06:31:12.460016 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/apiserver.crt -> /var/lib/minikube/certs/apiserver.crt
I0315 06:31:12.460025 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/apiserver.key -> /var/lib/minikube/certs/apiserver.key
I0315 06:31:12.460039 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/proxy-client.crt -> /var/lib/minikube/certs/proxy-client.crt
I0315 06:31:12.460056 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/proxy-client.key -> /var/lib/minikube/certs/proxy-client.key
I0315 06:31:12.460105 33437 certs.go:484] found cert: /home/jenkins/minikube-integration/18213-8825/.minikube/certs/16075.pem (1338 bytes)
W0315 06:31:12.460142 33437 certs.go:480] ignoring /home/jenkins/minikube-integration/18213-8825/.minikube/certs/16075_empty.pem, impossibly tiny 0 bytes
I0315 06:31:12.460148 33437 certs.go:484] found cert: /home/jenkins/minikube-integration/18213-8825/.minikube/certs/ca-key.pem (1679 bytes)
I0315 06:31:12.460168 33437 certs.go:484] found cert: /home/jenkins/minikube-integration/18213-8825/.minikube/certs/ca.pem (1078 bytes)
I0315 06:31:12.460186 33437 certs.go:484] found cert: /home/jenkins/minikube-integration/18213-8825/.minikube/certs/cert.pem (1123 bytes)
I0315 06:31:12.460202 33437 certs.go:484] found cert: /home/jenkins/minikube-integration/18213-8825/.minikube/certs/key.pem (1679 bytes)
I0315 06:31:12.460236 33437 certs.go:484] found cert: /home/jenkins/minikube-integration/18213-8825/.minikube/files/etc/ssl/certs/160752.pem (1708 bytes)
I0315 06:31:12.460264 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/ca.crt -> /usr/share/ca-certificates/minikubeCA.pem
I0315 06:31:12.460276 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/certs/16075.pem -> /usr/share/ca-certificates/16075.pem
I0315 06:31:12.460287 33437 vm_assets.go:163] NewFileAsset: /home/jenkins/minikube-integration/18213-8825/.minikube/files/etc/ssl/certs/160752.pem -> /usr/share/ca-certificates/160752.pem
I0315 06:31:12.460824 33437 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18213-8825/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0315 06:31:12.560429 33437 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18213-8825/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1679 bytes)
I0315 06:31:12.636236 33437 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18213-8825/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0315 06:31:12.785307 33437 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18213-8825/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1675 bytes)
I0315 06:31:12.994552 33437 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1432 bytes)
I0315 06:31:13.094315 33437 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1679 bytes)
I0315 06:31:13.148757 33437 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0315 06:31:13.186255 33437 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18213-8825/.minikube/profiles/ha-866665/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1679 bytes)
I0315 06:31:13.219468 33437 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18213-8825/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0315 06:31:13.298432 33437 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18213-8825/.minikube/certs/16075.pem --> /usr/share/ca-certificates/16075.pem (1338 bytes)
I0315 06:31:13.356395 33437 ssh_runner.go:362] scp /home/jenkins/minikube-integration/18213-8825/.minikube/files/etc/ssl/certs/160752.pem --> /usr/share/ca-certificates/160752.pem (1708 bytes)
I0315 06:31:13.388275 33437 ssh_runner.go:362] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0315 06:31:13.416436 33437 ssh_runner.go:195] Run: openssl version
I0315 06:31:13.425310 33437 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/16075.pem && ln -fs /usr/share/ca-certificates/16075.pem /etc/ssl/certs/16075.pem"
I0315 06:31:13.441059 33437 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/16075.pem
I0315 06:31:13.446108 33437 certs.go:528] hashing: -rw-r--r-- 1 root root 1338 Mar 15 06:06 /usr/share/ca-certificates/16075.pem
I0315 06:31:13.446176 33437 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/16075.pem
I0315 06:31:13.452449 33437 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/51391683.0 || ln -fs /etc/ssl/certs/16075.pem /etc/ssl/certs/51391683.0"
I0315 06:31:13.463777 33437 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/160752.pem && ln -fs /usr/share/ca-certificates/160752.pem /etc/ssl/certs/160752.pem"
I0315 06:31:13.475540 33437 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/160752.pem
I0315 06:31:13.480658 33437 certs.go:528] hashing: -rw-r--r-- 1 root root 1708 Mar 15 06:06 /usr/share/ca-certificates/160752.pem
I0315 06:31:13.480725 33437 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/160752.pem
I0315 06:31:13.490764 33437 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/3ec20f2e.0 || ln -fs /etc/ssl/certs/160752.pem /etc/ssl/certs/3ec20f2e.0"
I0315 06:31:13.507483 33437 ssh_runner.go:195] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0315 06:31:13.522259 33437 ssh_runner.go:195] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0315 06:31:13.527387 33437 certs.go:528] hashing: -rw-r--r-- 1 root root 1111 Mar 15 05:58 /usr/share/ca-certificates/minikubeCA.pem
I0315 06:31:13.527450 33437 ssh_runner.go:195] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0315 06:31:13.535340 33437 ssh_runner.go:195] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0315 06:31:13.547679 33437 ssh_runner.go:195] Run: stat /var/lib/minikube/certs/apiserver-kubelet-client.crt
I0315 06:31:13.555390 33437 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/apiserver-etcd-client.crt -checkend 86400
I0315 06:31:13.561596 33437 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/apiserver-kubelet-client.crt -checkend 86400
I0315 06:31:13.569097 33437 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/etcd/server.crt -checkend 86400
I0315 06:31:13.575180 33437 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/etcd/healthcheck-client.crt -checkend 86400
I0315 06:31:13.581211 33437 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/etcd/peer.crt -checkend 86400
I0315 06:31:13.589108 33437 ssh_runner.go:195] Run: openssl x509 -noout -in /var/lib/minikube/certs/front-proxy-client.crt -checkend 86400
I0315 06:31:13.597196 33437 kubeadm.go:391] StartCluster: {Name:ha-866665 KeepContext:false EmbedCerts:false MinikubeISO:https://storage.googleapis.com/minikube-builds/iso/18213/minikube-v1.32.1-1710459732-18213-amd64.iso KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.42-1710284843-18375@sha256:d67c38c9fc2ad14c48d95e17cbac49314325db5758d8f7b3de60b927e62ce94f Memory:2200 CPUs:2 DiskSize:20000 Driver:kvm2 HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.28.4 Clust
erName:ha-866665 Namespace:default APIServerHAVIP:192.168.39.254 APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:crio CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP:192.168.39.78 Port:8443 KubernetesVersion:v1.28.4 ContainerRuntime:crio ControlPlane:true Worker:true} {Name:m02 IP:192.168.39.27 Port:8443 KubernetesVersion:v1.28.4 ContainerRuntime:crio ControlPlane:true Worker:true} {Name:m04 IP:192.168.39.184 Port:0 KubernetesVersion:v1.28.4 ContainerRuntime:crio ControlPlane:false Worker:true}] Addons:map[ambassador:false auto-pause:false cloud-spanner:false csi-hostpath-driver:false dashboard:false default-storageclass:false efk:false freshpod:false gcp-auth:false gvisor:false headlamp:false helm-tiller:false inaccel:false ingress:false ingress-dns:false in
spektor-gadget:false istio:false istio-provisioner:false kong:false kubeflow:false kubevirt:false logviewer:false metallb:false metrics-server:false nvidia-device-plugin:false nvidia-driver-installer:false nvidia-gpu-device-plugin:false olm:false pod-security-policy:false portainer:false registry:false registry-aliases:false registry-creds:false storage-provisioner:false storage-provisioner-gluster:false storage-provisioner-rancher:false volumesnapshots:false yakd:false] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true apps_running:true default_sa:true extra:true kubelet:true node_ready:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:true ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/jenkins:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:fal
se DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0315 06:31:13.597364 33437 cri.go:54] listing CRI containers in root : {State:paused Name: Namespaces:[kube-system]}
I0315 06:31:13.597432 33437 ssh_runner.go:195] Run: sudo -s eval "crictl ps -a --quiet --label io.kubernetes.pod.namespace=kube-system"
I0315 06:31:13.654360 33437 cri.go:89] found id: "a3b8244e29a11ef4a5d7f6bc11533ad0fbe55bc470717991ed5d1537f8c04703"
I0315 06:31:13.654388 33437 cri.go:89] found id: "f8f276ed61ae487693a7e0d2b2f9aa737ceb4003ccb92006a179a894db6abb8a"
I0315 06:31:13.654393 33437 cri.go:89] found id: "4632db4347aeb5d329b75f46db342bda137407096006e2ba18e03e6a7ab005a2"
I0315 06:31:13.654401 33437 cri.go:89] found id: "53858589abe0969b0d43fd253c11bdba2cc0c2966139f79c07bbf78cffdcd869"
I0315 06:31:13.654406 33437 cri.go:89] found id: "bdde9d3309aa653bdde9bc5fb009352128cc082c6210723aabf3090316773af4"
I0315 06:31:13.654411 33437 cri.go:89] found id: "e09471036e57d5a30f82f0f6b4431734b99e26e0508edbf44bf1bb0772de441a"
I0315 06:31:13.654414 33437 cri.go:89] found id: "e4370ef8479c8071e7f6e97ceb9378aad8d67434c8fb0c47b8d0fc9bda7f3596"
I0315 06:31:13.654418 33437 cri.go:89] found id: "cb4635f3b41c27cc574f00181fa43463b7015da1cead64c27648cf9f11a76de7"
I0315 06:31:13.654422 33437 cri.go:89] found id: "5eaa4a539d19c14760c52937c8f1e1fe12532f9f8bd2064f645d738bcd019bcb"
I0315 06:31:13.654429 33437 cri.go:89] found id: "a632d3a2baa8574a6dfa2fd14d34d67ff1594473528173c4f3cfd95f4725dcbe"
I0315 06:31:13.654434 33437 cri.go:89] found id: "e490c56eb4c5d1eed5a5a3d95c47fd84a974b0db29822a120857edf9b3749306"
I0315 06:31:13.654438 33437 cri.go:89] found id: "c7b60409e5d22324ba312edc6c4f5e94d3a1b00f5d9f7dc2922756f23a23f31e"
I0315 06:31:13.654447 33437 cri.go:89] found id: "950153b4c9efe4316e3c3891bb3ef221780f0fe05967f5dd112e0b11f5c73088"
I0315 06:31:13.654451 33437 cri.go:89] found id: "a72cadbbcef74cbfe983b247bf865ecdf46f6ca8526deeb1657b9f137651e3f2"
I0315 06:31:13.654473 33437 cri.go:89] found id: "20a813e0950a0ee7b0095bc9f0aeae8c6e0eeac53471a0be98dbeaf701f36b8a"
I0315 06:31:13.654481 33437 cri.go:89] found id: "002360447d19f48a6c9aceda11e1ac67337580d0ec0bac7f9b75503f387efb0c"
I0315 06:31:13.654485 33437 cri.go:89] found id: "f1c6cdc2511cf629dab56a84f54e1f95515f47a5fe7d810e843ab351d4c4db1c"
I0315 06:31:13.654494 33437 cri.go:89] found id: "8507c4a363e13f76515845ccd38eb4fda5ddfbd17bb8eb5a14e138f2607603de"
I0315 06:31:13.654501 33437 cri.go:89] found id: "bede6c7f8912b13b1393e0fdd692ee76e9a0c621b35ebdaae1e0fdbc7d256780"
I0315 06:31:13.654506 33437 cri.go:89] found id: "c0ecd2e85889290ffc2876db9407cbc49c9213575c5c2edb96fba1d4a13b8c90"
I0315 06:31:13.654513 33437 cri.go:89] found id: "c07640cff4cedd7200db6dc04a3f4d74a7cb002877f61eb0dd86fb6c5e4c00d0"
I0315 06:31:13.654517 33437 cri.go:89] found id: "7fcd79ed43f7b877296fc88df84b18b419f9eee806ece61a2fb354f3b078d0c3"
I0315 06:31:13.654524 33437 cri.go:89] found id: "adc81452470007eeb06e6ad4a10ac0e477a9ed99489bb369e2bb63c888bec435"
I0315 06:31:13.654528 33437 cri.go:89] found id: ""
I0315 06:31:13.654578 33437 ssh_runner.go:195] Run: sudo runc list -f json
==> CRI-O <==
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.355766781Z" level=debug msg="Response: &ImageFsInfoResponse{ImageFilesystems:[]*FilesystemUsage{&FilesystemUsage{Timestamp:1710484538355736051,FsId:&FilesystemIdentifier{Mountpoint:/var/lib/containers/storage/overlay-images,},UsedBytes:&UInt64Value{Value:146620,},InodesUsed:&UInt64Value{Value:72,},},},ContainerFilesystems:[]*FilesystemUsage{},}" file="otel-collector/interceptors.go:74" id=ac074489-4fbd-4871-86cd-01f436fed1d1 name=/runtime.v1.ImageService/ImageFsInfo
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.356716179Z" level=debug msg="Request: &ListContainersRequest{Filter:&ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},},}" file="otel-collector/interceptors.go:62" id=575b06ed-b6de-4fed-9687-d5dca5840e95 name=/runtime.v1.RuntimeService/ListContainers
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.356771050Z" level=debug msg="No filters were applied, returning full container list" file="server/container_list.go:60" id=575b06ed-b6de-4fed-9687-d5dca5840e95 name=/runtime.v1.RuntimeService/ListContainers
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.357153136Z" level=debug msg="Response: &ListContainersResponse{Containers:[]*Container{&Container{Id:8f4c9a0644c94c4301a7d96a179f3c1225d5f24d9c926c09380f1e111322dfdd,PodSandboxId:16c817ba7d264b8f7bba3ab1d1ff3074904f7f53258453dc326eaf7fab00203f,Metadata:&ContainerMetadata{Name:kube-controller-manager,Attempt:5,},Image:&ImageSpec{Image:d058aa5ab969ce7b84d25e7188be1f80633b18db8ea7d02d9d0a78e676236591,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:d058aa5ab969ce7b84d25e7188be1f80633b18db8ea7d02d9d0a78e676236591,State:CONTAINER_EXITED,CreatedAt:1710484440561853992,Labels:map[string]string{io.kubernetes.container.name: kube-controller-manager,io.kubernetes.pod.name: kube-controller-manager-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: c0c36bd8aefb52162dc1bcef34f4bb25,},Annotations:map[string]string{io.kubernetes.container.hash: 4b9c51fc,io.kubernetes.container.restartCount: 5,io.kuber
netes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:8568bece5a49f6f0a84fb69937c117029e7dedd90c81050e5a36aef2a90dc985,PodSandboxId:1a1fdba5ec224a16b91ab05b40fb30f9872f97f1e31f57c3bc8180a2d08c9b0e,Metadata:&ContainerMetadata{Name:busybox,Attempt:2,},Image:&ImageSpec{Image:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,State:CONTAINER_RUNNING,CreatedAt:1710484306218979337,Labels:map[string]string{io.kubernetes.container.name: busybox,io.kubernetes.pod.name: busybox-5b5d89c9d6-82knb,io.kubernetes.pod.namespace: default,io.kubernetes.pod.uid: c12d72ab-189f-4a4a-a7df-54e10184a9ac,},Annotations:map[string]string{io.kubernetes.container.hash: 7b2cc69f,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessageP
ath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:10518fb395ccef0d12367b01f3a67613f203a91a536807a5036c49103719fc54,PodSandboxId:d7a69a3a337af71582ce1051a4f71e864ffa1af63897bfe9d000cccced5c310f,Metadata:&ContainerMetadata{Name:kindnet-cni,Attempt:5,},Image:&ImageSpec{Image:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5,State:CONTAINER_EXITED,CreatedAt:1710484278390877828,Labels:map[string]string{io.kubernetes.container.name: kindnet-cni,io.kubernetes.pod.name: kindnet-9nvvx,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 4c5333df-bb98-4f27-9197-875a160f4ff6,},Annotations:map[string]string{io.kubernetes.container.hash: e5c72ddc,io.kubernetes.container.restartCount: 5,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubern
etes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:fd937a8a91decc1f378cd56d4068b095fdceb3673f429ed113e2508c22d3d4a5,PodSandboxId:9f2fb2d671096660329cda96fc359a94e105ee6b24bfcbbfca16d6e253ac9735,Metadata:&ContainerMetadata{Name:kube-proxy,Attempt:2,},Image:&ImageSpec{Image:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,State:CONTAINER_RUNNING,CreatedAt:1710484278257874276,Labels:map[string]string{io.kubernetes.container.name: kube-proxy,io.kubernetes.pod.name: kube-proxy-sbxgg,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 33fac82d-5f3a-42b8-99b7-1f4ee45c0f98,},Annotations:map[string]string{io.kubernetes.container.hash: 2d71c9ea,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessage
Policy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:f31b9e9704e22843a1d46c9994b8fad60de501458b0915ca32d41c10c0f1bde4,PodSandboxId:a0a76006f9e8a838dea8efb6b347aa0d186f4a862312b796d3daefd54f80de2c,Metadata:&ContainerMetadata{Name:storage-provisioner,Attempt:6,},Image:&ImageSpec{Image:6e38f40d628db3002f5617342c8872c935de530d867d0f709a2fbda1a302a562,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:6e38f40d628db3002f5617342c8872c935de530d867d0f709a2fbda1a302a562,State:CONTAINER_EXITED,CreatedAt:1710484278208014606,Labels:map[string]string{io.kubernetes.container.name: storage-provisioner,io.kubernetes.pod.name: storage-provisioner,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: b11128b3-f84e-4526-992d-56e278c3f7c9,},Annotations:map[string]string{io.kubernetes.container.hash: 136548f5,io.kubernetes.container.restartCount: 6,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,
io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:959c3adf756ac670f71e9a23a6963729f6c9b1a1866a5044f8cc579dc7a28be8,PodSandboxId:22072600a839b56ec14de1ab04f264d93721072ea15b5f022bfd1fa67aa1a744,Metadata:&ContainerMetadata{Name:etcd,Attempt:2,},Image:&ImageSpec{Image:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,State:CONTAINER_RUNNING,CreatedAt:1710484278111817672,Labels:map[string]string{io.kubernetes.container.name: etcd,io.kubernetes.pod.name: etcd-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 72b65491f907cd3dd68444500227632e,},Annotations:map[string]string{io.kubernetes.container.hash: ad0ffc58,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&C
ontainer{Id:a3b8244e29a11ef4a5d7f6bc11533ad0fbe55bc470717991ed5d1537f8c04703,PodSandboxId:cf57c2ff9f3b2fd48a06472a048c05a9853e0d41b8f41290305af04765eca0fa,Metadata:&ContainerMetadata{Name:coredns,Attempt:2,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_RUNNING,CreatedAt:1710484272859108005,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-r57px,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 4d7f8905-5519-453d-a9a3-26b5d511f1c3,},Annotations:map[string]string{io.kubernetes.container.hash: 579e5097,io.kubernetes.container.ports: [{\"name\":\"dns\",\"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.contai
ner.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:f8f276ed61ae487693a7e0d2b2f9aa737ceb4003ccb92006a179a894db6abb8a,PodSandboxId:14af170de4b571898874a48dcf002fe94fb21ebca6d8b0ad7fe8f5a1da7e1810,Metadata:&ContainerMetadata{Name:coredns,Attempt:2,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_RUNNING,CreatedAt:1710484272801262973,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-mgthb,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 6498160a-372c-4273-a82c-b06c4b7b239b,},Annotations:map[string]string{io.kubernetes.container.hash: 1b336595,io.kubernetes.container.ports: [{\"name\":\"dns\",\
"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:4632db4347aeb5d329b75f46db342bda137407096006e2ba18e03e6a7ab005a2,PodSandboxId:09c39329a8da2573ad3cbdd895668b0e9a7645b64732383eae88272e9f00894b,Metadata:&ContainerMetadata{Name:kube-vip,Attempt:5,},Image:&ImageSpec{Image:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,State:CONTAINER_RUNNING,CreatedAt:1710484272726972441,Labels:map[string]string{io.kubernetes.container.name: kube-vip,io.kubernetes.pod.name: kube-vip-ha-866665,io.kubernetes.pod.namespace: kub
e-system,io.kubernetes.pod.uid: affdbe5d0709ec0c8cfe4e796df74130,},Annotations:map[string]string{io.kubernetes.container.hash: d7e5eb98,io.kubernetes.container.restartCount: 5,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:53858589abe0969b0d43fd253c11bdba2cc0c2966139f79c07bbf78cffdcd869,PodSandboxId:7f2d26260bc132df83346e260059f600807bc71e82449c959512f6772a42dc78,Metadata:&ContainerMetadata{Name:kube-scheduler,Attempt:2,},Image:&ImageSpec{Image:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,State:CONTAINER_RUNNING,CreatedAt:1710484272697875290,Labels:map[string]string{io.kubernetes.container.name: kube-scheduler,io.kubernetes.pod.name: kube-scheduler-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernete
s.pod.uid: 06e4fc01158af0abcc722b76c4fcbaee,},Annotations:map[string]string{io.kubernetes.container.hash: e1639c7a,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:e4370ef8479c8071e7f6e97ceb9378aad8d67434c8fb0c47b8d0fc9bda7f3596,PodSandboxId:f7b655acbd70858956764f6cbf8b2cb467f44a747f8c4a0a4fa876282cf4b56a,Metadata:&ContainerMetadata{Name:kube-apiserver,Attempt:4,},Image:&ImageSpec{Image:7fe0e6f37db33464725e616a12ccc4e36970370005a2b09683a974db6350c257,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:7fe0e6f37db33464725e616a12ccc4e36970370005a2b09683a974db6350c257,State:CONTAINER_EXITED,CreatedAt:1710484012956974785,Labels:map[string]string{io.kubernetes.container.name: kube-apiserver,io.kubernetes.pod.name: kube-apiserver-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: ec32969267e
5d443d53332f70d668161,},Annotations:map[string]string{io.kubernetes.container.hash: 4d761a02,io.kubernetes.container.restartCount: 4,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:cb4635f3b41c27cc574f00181fa43463b7015da1cead64c27648cf9f11a76de7,PodSandboxId:b3fef0e73d7bb1217924a1358207053526d7b678fa6c4f039b71d9805717c222,Metadata:&ContainerMetadata{Name:kube-vip,Attempt:4,},Image:&ImageSpec{Image:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,State:CONTAINER_EXITED,CreatedAt:1710483971559144395,Labels:map[string]string{io.kubernetes.container.name: kube-vip,io.kubernetes.pod.name: kube-vip-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: affdbe5d0709ec0c8cfe4e796df74130,},Annotations:map[
string]string{io.kubernetes.container.hash: d7e5eb98,io.kubernetes.container.restartCount: 4,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:d0b44fa63b7f7d4b61990c88d7da99f2d4fb738670e3e6e22354b5a5f3a0c29d,PodSandboxId:344f655b58490a4f777d55dd2c551b578c705e7b4a71a6577daa53c114bbc4a6,Metadata:&ContainerMetadata{Name:busybox,Attempt:1,},Image:&ImageSpec{Image:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,State:CONTAINER_EXITED,CreatedAt:1710483786887352054,Labels:map[string]string{io.kubernetes.container.name: busybox,io.kubernetes.pod.name: busybox-5b5d89c9d6-82knb,io.kubernetes.pod.namespace: default,io.kubernetes.pod.uid: c12d72ab-189f-4a4a-a7df-54e10184a9ac,},Annotations:map[string]string{io.kubernetes.containe
r.hash: 7b2cc69f,io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:a72cadbbcef74cbfe983b247bf865ecdf46f6ca8526deeb1657b9f137651e3f2,PodSandboxId:c28c4fa9bcc72d9300f656805a3e97c6dcdddab3671e12d0480e6e96be9c6391,Metadata:&ContainerMetadata{Name:kube-proxy,Attempt:1,},Image:&ImageSpec{Image:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,State:CONTAINER_EXITED,CreatedAt:1710483753567192605,Labels:map[string]string{io.kubernetes.container.name: kube-proxy,io.kubernetes.pod.name: kube-proxy-sbxgg,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 33fac82d-5f3a-42b8-99b7-1f4ee45c0f98,},Annotations:map[string]string{io.kubernetes.container.hash: 2d71c9ea,io.kubernetes.con
tainer.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:c7b60409e5d22324ba312edc6c4f5e94d3a1b00f5d9f7dc2922756f23a23f31e,PodSandboxId:268a2a5390630a207b2375a2133c6867d83efaacce2f2dcb1508f1a9aaf65799,Metadata:&ContainerMetadata{Name:coredns,Attempt:1,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_EXITED,CreatedAt:1710483753601335090,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-r57px,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 4d7f8905-5519-453d-a9a3-26b5d511f1c3,},Annotations:map[string]string{io.kubernetes.container.hash: 579e5097,io.kubernetes.container.ports: [{\"name\":\"dns\"
,\"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:f1c6cdc2511cf629dab56a84f54e1f95515f47a5fe7d810e843ab351d4c4db1c,PodSandboxId:8899126815da59ab432c51df4ada6cd869277719ab23e03fadd977ebca310687,Metadata:&ContainerMetadata{Name:kube-scheduler,Attempt:1,},Image:&ImageSpec{Image:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,State:CONTAINER_EXITED,CreatedAt:1710483753499514985,Labels:map[string]string{io.kubernetes.container.name: kube-scheduler,io.kubernetes.pod.name: kube-scheduler-ha-866665,io.kubernetes
.pod.namespace: kube-system,io.kubernetes.pod.uid: 06e4fc01158af0abcc722b76c4fcbaee,},Annotations:map[string]string{io.kubernetes.container.hash: e1639c7a,io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:20a813e0950a0ee7b0095bc9f0aeae8c6e0eeac53471a0be98dbeaf701f36b8a,PodSandboxId:79337bac30908f0c1e437fa581563a038ff9e4cfd59fdd43f1d02020f0cac259,Metadata:&ContainerMetadata{Name:etcd,Attempt:1,},Image:&ImageSpec{Image:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,State:CONTAINER_EXITED,CreatedAt:1710483753556829494,Labels:map[string]string{io.kubernetes.container.name: etcd,io.kubernetes.pod.name: etcd-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 7
2b65491f907cd3dd68444500227632e,},Annotations:map[string]string{io.kubernetes.container.hash: ad0ffc58,io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:8507c4a363e13f76515845ccd38eb4fda5ddfbd17bb8eb5a14e138f2607603de,PodSandboxId:a1032069bcb086a4d107a66ed66790e1c54fe03538931ef9a0f95200989338b1,Metadata:&ContainerMetadata{Name:coredns,Attempt:1,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_EXITED,CreatedAt:1710483748541541438,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-mgthb,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 6498160a-372c-4273-a82c-b06c4b7b239b,
},Annotations:map[string]string{io.kubernetes.container.hash: 1b336595,io.kubernetes.container.ports: [{\"name\":\"dns\",\"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},},}" file="otel-collector/interceptors.go:74" id=575b06ed-b6de-4fed-9687-d5dca5840e95 name=/runtime.v1.RuntimeService/ListContainers
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.411099815Z" level=debug msg="Request: &VersionRequest{Version:,}" file="otel-collector/interceptors.go:62" id=2bc95374-3be8-4924-989a-6555a0bcb8e0 name=/runtime.v1.RuntimeService/Version
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.411174974Z" level=debug msg="Response: &VersionResponse{Version:0.1.0,RuntimeName:cri-o,RuntimeVersion:1.29.1,RuntimeApiVersion:v1,}" file="otel-collector/interceptors.go:74" id=2bc95374-3be8-4924-989a-6555a0bcb8e0 name=/runtime.v1.RuntimeService/Version
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.412979452Z" level=debug msg="Request: &ImageFsInfoRequest{}" file="otel-collector/interceptors.go:62" id=19338065-ce3d-4668-9d79-6be33d5ab5bb name=/runtime.v1.ImageService/ImageFsInfo
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.413514898Z" level=debug msg="Response: &ImageFsInfoResponse{ImageFilesystems:[]*FilesystemUsage{&FilesystemUsage{Timestamp:1710484538413491178,FsId:&FilesystemIdentifier{Mountpoint:/var/lib/containers/storage/overlay-images,},UsedBytes:&UInt64Value{Value:146620,},InodesUsed:&UInt64Value{Value:72,},},},ContainerFilesystems:[]*FilesystemUsage{},}" file="otel-collector/interceptors.go:74" id=19338065-ce3d-4668-9d79-6be33d5ab5bb name=/runtime.v1.ImageService/ImageFsInfo
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.414057488Z" level=debug msg="Request: &ListContainersRequest{Filter:&ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},},}" file="otel-collector/interceptors.go:62" id=1e64d9bd-54aa-43d1-92d1-d1f470bec160 name=/runtime.v1.RuntimeService/ListContainers
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.414113850Z" level=debug msg="No filters were applied, returning full container list" file="server/container_list.go:60" id=1e64d9bd-54aa-43d1-92d1-d1f470bec160 name=/runtime.v1.RuntimeService/ListContainers
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.414583093Z" level=debug msg="Response: &ListContainersResponse{Containers:[]*Container{&Container{Id:8f4c9a0644c94c4301a7d96a179f3c1225d5f24d9c926c09380f1e111322dfdd,PodSandboxId:16c817ba7d264b8f7bba3ab1d1ff3074904f7f53258453dc326eaf7fab00203f,Metadata:&ContainerMetadata{Name:kube-controller-manager,Attempt:5,},Image:&ImageSpec{Image:d058aa5ab969ce7b84d25e7188be1f80633b18db8ea7d02d9d0a78e676236591,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:d058aa5ab969ce7b84d25e7188be1f80633b18db8ea7d02d9d0a78e676236591,State:CONTAINER_EXITED,CreatedAt:1710484440561853992,Labels:map[string]string{io.kubernetes.container.name: kube-controller-manager,io.kubernetes.pod.name: kube-controller-manager-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: c0c36bd8aefb52162dc1bcef34f4bb25,},Annotations:map[string]string{io.kubernetes.container.hash: 4b9c51fc,io.kubernetes.container.restartCount: 5,io.kuber
netes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:8568bece5a49f6f0a84fb69937c117029e7dedd90c81050e5a36aef2a90dc985,PodSandboxId:1a1fdba5ec224a16b91ab05b40fb30f9872f97f1e31f57c3bc8180a2d08c9b0e,Metadata:&ContainerMetadata{Name:busybox,Attempt:2,},Image:&ImageSpec{Image:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,State:CONTAINER_RUNNING,CreatedAt:1710484306218979337,Labels:map[string]string{io.kubernetes.container.name: busybox,io.kubernetes.pod.name: busybox-5b5d89c9d6-82knb,io.kubernetes.pod.namespace: default,io.kubernetes.pod.uid: c12d72ab-189f-4a4a-a7df-54e10184a9ac,},Annotations:map[string]string{io.kubernetes.container.hash: 7b2cc69f,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessageP
ath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:10518fb395ccef0d12367b01f3a67613f203a91a536807a5036c49103719fc54,PodSandboxId:d7a69a3a337af71582ce1051a4f71e864ffa1af63897bfe9d000cccced5c310f,Metadata:&ContainerMetadata{Name:kindnet-cni,Attempt:5,},Image:&ImageSpec{Image:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5,State:CONTAINER_EXITED,CreatedAt:1710484278390877828,Labels:map[string]string{io.kubernetes.container.name: kindnet-cni,io.kubernetes.pod.name: kindnet-9nvvx,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 4c5333df-bb98-4f27-9197-875a160f4ff6,},Annotations:map[string]string{io.kubernetes.container.hash: e5c72ddc,io.kubernetes.container.restartCount: 5,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubern
etes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:fd937a8a91decc1f378cd56d4068b095fdceb3673f429ed113e2508c22d3d4a5,PodSandboxId:9f2fb2d671096660329cda96fc359a94e105ee6b24bfcbbfca16d6e253ac9735,Metadata:&ContainerMetadata{Name:kube-proxy,Attempt:2,},Image:&ImageSpec{Image:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,State:CONTAINER_RUNNING,CreatedAt:1710484278257874276,Labels:map[string]string{io.kubernetes.container.name: kube-proxy,io.kubernetes.pod.name: kube-proxy-sbxgg,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 33fac82d-5f3a-42b8-99b7-1f4ee45c0f98,},Annotations:map[string]string{io.kubernetes.container.hash: 2d71c9ea,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessage
Policy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:f31b9e9704e22843a1d46c9994b8fad60de501458b0915ca32d41c10c0f1bde4,PodSandboxId:a0a76006f9e8a838dea8efb6b347aa0d186f4a862312b796d3daefd54f80de2c,Metadata:&ContainerMetadata{Name:storage-provisioner,Attempt:6,},Image:&ImageSpec{Image:6e38f40d628db3002f5617342c8872c935de530d867d0f709a2fbda1a302a562,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:6e38f40d628db3002f5617342c8872c935de530d867d0f709a2fbda1a302a562,State:CONTAINER_EXITED,CreatedAt:1710484278208014606,Labels:map[string]string{io.kubernetes.container.name: storage-provisioner,io.kubernetes.pod.name: storage-provisioner,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: b11128b3-f84e-4526-992d-56e278c3f7c9,},Annotations:map[string]string{io.kubernetes.container.hash: 136548f5,io.kubernetes.container.restartCount: 6,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,
io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:959c3adf756ac670f71e9a23a6963729f6c9b1a1866a5044f8cc579dc7a28be8,PodSandboxId:22072600a839b56ec14de1ab04f264d93721072ea15b5f022bfd1fa67aa1a744,Metadata:&ContainerMetadata{Name:etcd,Attempt:2,},Image:&ImageSpec{Image:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,State:CONTAINER_RUNNING,CreatedAt:1710484278111817672,Labels:map[string]string{io.kubernetes.container.name: etcd,io.kubernetes.pod.name: etcd-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 72b65491f907cd3dd68444500227632e,},Annotations:map[string]string{io.kubernetes.container.hash: ad0ffc58,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&C
ontainer{Id:a3b8244e29a11ef4a5d7f6bc11533ad0fbe55bc470717991ed5d1537f8c04703,PodSandboxId:cf57c2ff9f3b2fd48a06472a048c05a9853e0d41b8f41290305af04765eca0fa,Metadata:&ContainerMetadata{Name:coredns,Attempt:2,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_RUNNING,CreatedAt:1710484272859108005,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-r57px,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 4d7f8905-5519-453d-a9a3-26b5d511f1c3,},Annotations:map[string]string{io.kubernetes.container.hash: 579e5097,io.kubernetes.container.ports: [{\"name\":\"dns\",\"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.contai
ner.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:f8f276ed61ae487693a7e0d2b2f9aa737ceb4003ccb92006a179a894db6abb8a,PodSandboxId:14af170de4b571898874a48dcf002fe94fb21ebca6d8b0ad7fe8f5a1da7e1810,Metadata:&ContainerMetadata{Name:coredns,Attempt:2,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_RUNNING,CreatedAt:1710484272801262973,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-mgthb,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 6498160a-372c-4273-a82c-b06c4b7b239b,},Annotations:map[string]string{io.kubernetes.container.hash: 1b336595,io.kubernetes.container.ports: [{\"name\":\"dns\",\
"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:4632db4347aeb5d329b75f46db342bda137407096006e2ba18e03e6a7ab005a2,PodSandboxId:09c39329a8da2573ad3cbdd895668b0e9a7645b64732383eae88272e9f00894b,Metadata:&ContainerMetadata{Name:kube-vip,Attempt:5,},Image:&ImageSpec{Image:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,State:CONTAINER_RUNNING,CreatedAt:1710484272726972441,Labels:map[string]string{io.kubernetes.container.name: kube-vip,io.kubernetes.pod.name: kube-vip-ha-866665,io.kubernetes.pod.namespace: kub
e-system,io.kubernetes.pod.uid: affdbe5d0709ec0c8cfe4e796df74130,},Annotations:map[string]string{io.kubernetes.container.hash: d7e5eb98,io.kubernetes.container.restartCount: 5,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:53858589abe0969b0d43fd253c11bdba2cc0c2966139f79c07bbf78cffdcd869,PodSandboxId:7f2d26260bc132df83346e260059f600807bc71e82449c959512f6772a42dc78,Metadata:&ContainerMetadata{Name:kube-scheduler,Attempt:2,},Image:&ImageSpec{Image:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,State:CONTAINER_RUNNING,CreatedAt:1710484272697875290,Labels:map[string]string{io.kubernetes.container.name: kube-scheduler,io.kubernetes.pod.name: kube-scheduler-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernete
s.pod.uid: 06e4fc01158af0abcc722b76c4fcbaee,},Annotations:map[string]string{io.kubernetes.container.hash: e1639c7a,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:e4370ef8479c8071e7f6e97ceb9378aad8d67434c8fb0c47b8d0fc9bda7f3596,PodSandboxId:f7b655acbd70858956764f6cbf8b2cb467f44a747f8c4a0a4fa876282cf4b56a,Metadata:&ContainerMetadata{Name:kube-apiserver,Attempt:4,},Image:&ImageSpec{Image:7fe0e6f37db33464725e616a12ccc4e36970370005a2b09683a974db6350c257,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:7fe0e6f37db33464725e616a12ccc4e36970370005a2b09683a974db6350c257,State:CONTAINER_EXITED,CreatedAt:1710484012956974785,Labels:map[string]string{io.kubernetes.container.name: kube-apiserver,io.kubernetes.pod.name: kube-apiserver-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: ec32969267e
5d443d53332f70d668161,},Annotations:map[string]string{io.kubernetes.container.hash: 4d761a02,io.kubernetes.container.restartCount: 4,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:cb4635f3b41c27cc574f00181fa43463b7015da1cead64c27648cf9f11a76de7,PodSandboxId:b3fef0e73d7bb1217924a1358207053526d7b678fa6c4f039b71d9805717c222,Metadata:&ContainerMetadata{Name:kube-vip,Attempt:4,},Image:&ImageSpec{Image:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,State:CONTAINER_EXITED,CreatedAt:1710483971559144395,Labels:map[string]string{io.kubernetes.container.name: kube-vip,io.kubernetes.pod.name: kube-vip-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: affdbe5d0709ec0c8cfe4e796df74130,},Annotations:map[
string]string{io.kubernetes.container.hash: d7e5eb98,io.kubernetes.container.restartCount: 4,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:d0b44fa63b7f7d4b61990c88d7da99f2d4fb738670e3e6e22354b5a5f3a0c29d,PodSandboxId:344f655b58490a4f777d55dd2c551b578c705e7b4a71a6577daa53c114bbc4a6,Metadata:&ContainerMetadata{Name:busybox,Attempt:1,},Image:&ImageSpec{Image:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,State:CONTAINER_EXITED,CreatedAt:1710483786887352054,Labels:map[string]string{io.kubernetes.container.name: busybox,io.kubernetes.pod.name: busybox-5b5d89c9d6-82knb,io.kubernetes.pod.namespace: default,io.kubernetes.pod.uid: c12d72ab-189f-4a4a-a7df-54e10184a9ac,},Annotations:map[string]string{io.kubernetes.containe
r.hash: 7b2cc69f,io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:a72cadbbcef74cbfe983b247bf865ecdf46f6ca8526deeb1657b9f137651e3f2,PodSandboxId:c28c4fa9bcc72d9300f656805a3e97c6dcdddab3671e12d0480e6e96be9c6391,Metadata:&ContainerMetadata{Name:kube-proxy,Attempt:1,},Image:&ImageSpec{Image:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,State:CONTAINER_EXITED,CreatedAt:1710483753567192605,Labels:map[string]string{io.kubernetes.container.name: kube-proxy,io.kubernetes.pod.name: kube-proxy-sbxgg,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 33fac82d-5f3a-42b8-99b7-1f4ee45c0f98,},Annotations:map[string]string{io.kubernetes.container.hash: 2d71c9ea,io.kubernetes.con
tainer.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:c7b60409e5d22324ba312edc6c4f5e94d3a1b00f5d9f7dc2922756f23a23f31e,PodSandboxId:268a2a5390630a207b2375a2133c6867d83efaacce2f2dcb1508f1a9aaf65799,Metadata:&ContainerMetadata{Name:coredns,Attempt:1,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_EXITED,CreatedAt:1710483753601335090,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-r57px,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 4d7f8905-5519-453d-a9a3-26b5d511f1c3,},Annotations:map[string]string{io.kubernetes.container.hash: 579e5097,io.kubernetes.container.ports: [{\"name\":\"dns\"
,\"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:f1c6cdc2511cf629dab56a84f54e1f95515f47a5fe7d810e843ab351d4c4db1c,PodSandboxId:8899126815da59ab432c51df4ada6cd869277719ab23e03fadd977ebca310687,Metadata:&ContainerMetadata{Name:kube-scheduler,Attempt:1,},Image:&ImageSpec{Image:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,State:CONTAINER_EXITED,CreatedAt:1710483753499514985,Labels:map[string]string{io.kubernetes.container.name: kube-scheduler,io.kubernetes.pod.name: kube-scheduler-ha-866665,io.kubernetes
.pod.namespace: kube-system,io.kubernetes.pod.uid: 06e4fc01158af0abcc722b76c4fcbaee,},Annotations:map[string]string{io.kubernetes.container.hash: e1639c7a,io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:20a813e0950a0ee7b0095bc9f0aeae8c6e0eeac53471a0be98dbeaf701f36b8a,PodSandboxId:79337bac30908f0c1e437fa581563a038ff9e4cfd59fdd43f1d02020f0cac259,Metadata:&ContainerMetadata{Name:etcd,Attempt:1,},Image:&ImageSpec{Image:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,State:CONTAINER_EXITED,CreatedAt:1710483753556829494,Labels:map[string]string{io.kubernetes.container.name: etcd,io.kubernetes.pod.name: etcd-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 7
2b65491f907cd3dd68444500227632e,},Annotations:map[string]string{io.kubernetes.container.hash: ad0ffc58,io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:8507c4a363e13f76515845ccd38eb4fda5ddfbd17bb8eb5a14e138f2607603de,PodSandboxId:a1032069bcb086a4d107a66ed66790e1c54fe03538931ef9a0f95200989338b1,Metadata:&ContainerMetadata{Name:coredns,Attempt:1,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_EXITED,CreatedAt:1710483748541541438,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-mgthb,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 6498160a-372c-4273-a82c-b06c4b7b239b,
},Annotations:map[string]string{io.kubernetes.container.hash: 1b336595,io.kubernetes.container.ports: [{\"name\":\"dns\",\"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},},}" file="otel-collector/interceptors.go:74" id=1e64d9bd-54aa-43d1-92d1-d1f470bec160 name=/runtime.v1.RuntimeService/ListContainers
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.461373132Z" level=debug msg="Request: &VersionRequest{Version:,}" file="otel-collector/interceptors.go:62" id=38c244a1-cf73-4cde-8d56-58ec58c96cf3 name=/runtime.v1.RuntimeService/Version
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.461468490Z" level=debug msg="Response: &VersionResponse{Version:0.1.0,RuntimeName:cri-o,RuntimeVersion:1.29.1,RuntimeApiVersion:v1,}" file="otel-collector/interceptors.go:74" id=38c244a1-cf73-4cde-8d56-58ec58c96cf3 name=/runtime.v1.RuntimeService/Version
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.463540051Z" level=debug msg="Request: &ImageFsInfoRequest{}" file="otel-collector/interceptors.go:62" id=81d2dea2-c958-4e3e-b2aa-f64e7493d617 name=/runtime.v1.ImageService/ImageFsInfo
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.464714059Z" level=debug msg="Response: &ImageFsInfoResponse{ImageFilesystems:[]*FilesystemUsage{&FilesystemUsage{Timestamp:1710484538464682749,FsId:&FilesystemIdentifier{Mountpoint:/var/lib/containers/storage/overlay-images,},UsedBytes:&UInt64Value{Value:146620,},InodesUsed:&UInt64Value{Value:72,},},},ContainerFilesystems:[]*FilesystemUsage{},}" file="otel-collector/interceptors.go:74" id=81d2dea2-c958-4e3e-b2aa-f64e7493d617 name=/runtime.v1.ImageService/ImageFsInfo
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.465382057Z" level=debug msg="Request: &ListContainersRequest{Filter:&ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},},}" file="otel-collector/interceptors.go:62" id=2df5f215-e10d-48ea-a75d-b294ca76eaaa name=/runtime.v1.RuntimeService/ListContainers
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.465444038Z" level=debug msg="No filters were applied, returning full container list" file="server/container_list.go:60" id=2df5f215-e10d-48ea-a75d-b294ca76eaaa name=/runtime.v1.RuntimeService/ListContainers
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.465810222Z" level=debug msg="Response: &ListContainersResponse{Containers:[]*Container{&Container{Id:8f4c9a0644c94c4301a7d96a179f3c1225d5f24d9c926c09380f1e111322dfdd,PodSandboxId:16c817ba7d264b8f7bba3ab1d1ff3074904f7f53258453dc326eaf7fab00203f,Metadata:&ContainerMetadata{Name:kube-controller-manager,Attempt:5,},Image:&ImageSpec{Image:d058aa5ab969ce7b84d25e7188be1f80633b18db8ea7d02d9d0a78e676236591,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:d058aa5ab969ce7b84d25e7188be1f80633b18db8ea7d02d9d0a78e676236591,State:CONTAINER_EXITED,CreatedAt:1710484440561853992,Labels:map[string]string{io.kubernetes.container.name: kube-controller-manager,io.kubernetes.pod.name: kube-controller-manager-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: c0c36bd8aefb52162dc1bcef34f4bb25,},Annotations:map[string]string{io.kubernetes.container.hash: 4b9c51fc,io.kubernetes.container.restartCount: 5,io.kuber
netes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:8568bece5a49f6f0a84fb69937c117029e7dedd90c81050e5a36aef2a90dc985,PodSandboxId:1a1fdba5ec224a16b91ab05b40fb30f9872f97f1e31f57c3bc8180a2d08c9b0e,Metadata:&ContainerMetadata{Name:busybox,Attempt:2,},Image:&ImageSpec{Image:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,State:CONTAINER_RUNNING,CreatedAt:1710484306218979337,Labels:map[string]string{io.kubernetes.container.name: busybox,io.kubernetes.pod.name: busybox-5b5d89c9d6-82knb,io.kubernetes.pod.namespace: default,io.kubernetes.pod.uid: c12d72ab-189f-4a4a-a7df-54e10184a9ac,},Annotations:map[string]string{io.kubernetes.container.hash: 7b2cc69f,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessageP
ath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:10518fb395ccef0d12367b01f3a67613f203a91a536807a5036c49103719fc54,PodSandboxId:d7a69a3a337af71582ce1051a4f71e864ffa1af63897bfe9d000cccced5c310f,Metadata:&ContainerMetadata{Name:kindnet-cni,Attempt:5,},Image:&ImageSpec{Image:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5,State:CONTAINER_EXITED,CreatedAt:1710484278390877828,Labels:map[string]string{io.kubernetes.container.name: kindnet-cni,io.kubernetes.pod.name: kindnet-9nvvx,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 4c5333df-bb98-4f27-9197-875a160f4ff6,},Annotations:map[string]string{io.kubernetes.container.hash: e5c72ddc,io.kubernetes.container.restartCount: 5,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubern
etes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:fd937a8a91decc1f378cd56d4068b095fdceb3673f429ed113e2508c22d3d4a5,PodSandboxId:9f2fb2d671096660329cda96fc359a94e105ee6b24bfcbbfca16d6e253ac9735,Metadata:&ContainerMetadata{Name:kube-proxy,Attempt:2,},Image:&ImageSpec{Image:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,State:CONTAINER_RUNNING,CreatedAt:1710484278257874276,Labels:map[string]string{io.kubernetes.container.name: kube-proxy,io.kubernetes.pod.name: kube-proxy-sbxgg,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 33fac82d-5f3a-42b8-99b7-1f4ee45c0f98,},Annotations:map[string]string{io.kubernetes.container.hash: 2d71c9ea,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessage
Policy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:f31b9e9704e22843a1d46c9994b8fad60de501458b0915ca32d41c10c0f1bde4,PodSandboxId:a0a76006f9e8a838dea8efb6b347aa0d186f4a862312b796d3daefd54f80de2c,Metadata:&ContainerMetadata{Name:storage-provisioner,Attempt:6,},Image:&ImageSpec{Image:6e38f40d628db3002f5617342c8872c935de530d867d0f709a2fbda1a302a562,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:6e38f40d628db3002f5617342c8872c935de530d867d0f709a2fbda1a302a562,State:CONTAINER_EXITED,CreatedAt:1710484278208014606,Labels:map[string]string{io.kubernetes.container.name: storage-provisioner,io.kubernetes.pod.name: storage-provisioner,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: b11128b3-f84e-4526-992d-56e278c3f7c9,},Annotations:map[string]string{io.kubernetes.container.hash: 136548f5,io.kubernetes.container.restartCount: 6,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,
io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:959c3adf756ac670f71e9a23a6963729f6c9b1a1866a5044f8cc579dc7a28be8,PodSandboxId:22072600a839b56ec14de1ab04f264d93721072ea15b5f022bfd1fa67aa1a744,Metadata:&ContainerMetadata{Name:etcd,Attempt:2,},Image:&ImageSpec{Image:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,State:CONTAINER_RUNNING,CreatedAt:1710484278111817672,Labels:map[string]string{io.kubernetes.container.name: etcd,io.kubernetes.pod.name: etcd-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 72b65491f907cd3dd68444500227632e,},Annotations:map[string]string{io.kubernetes.container.hash: ad0ffc58,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&C
ontainer{Id:a3b8244e29a11ef4a5d7f6bc11533ad0fbe55bc470717991ed5d1537f8c04703,PodSandboxId:cf57c2ff9f3b2fd48a06472a048c05a9853e0d41b8f41290305af04765eca0fa,Metadata:&ContainerMetadata{Name:coredns,Attempt:2,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_RUNNING,CreatedAt:1710484272859108005,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-r57px,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 4d7f8905-5519-453d-a9a3-26b5d511f1c3,},Annotations:map[string]string{io.kubernetes.container.hash: 579e5097,io.kubernetes.container.ports: [{\"name\":\"dns\",\"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.contai
ner.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:f8f276ed61ae487693a7e0d2b2f9aa737ceb4003ccb92006a179a894db6abb8a,PodSandboxId:14af170de4b571898874a48dcf002fe94fb21ebca6d8b0ad7fe8f5a1da7e1810,Metadata:&ContainerMetadata{Name:coredns,Attempt:2,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_RUNNING,CreatedAt:1710484272801262973,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-mgthb,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 6498160a-372c-4273-a82c-b06c4b7b239b,},Annotations:map[string]string{io.kubernetes.container.hash: 1b336595,io.kubernetes.container.ports: [{\"name\":\"dns\",\
"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:4632db4347aeb5d329b75f46db342bda137407096006e2ba18e03e6a7ab005a2,PodSandboxId:09c39329a8da2573ad3cbdd895668b0e9a7645b64732383eae88272e9f00894b,Metadata:&ContainerMetadata{Name:kube-vip,Attempt:5,},Image:&ImageSpec{Image:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,State:CONTAINER_RUNNING,CreatedAt:1710484272726972441,Labels:map[string]string{io.kubernetes.container.name: kube-vip,io.kubernetes.pod.name: kube-vip-ha-866665,io.kubernetes.pod.namespace: kub
e-system,io.kubernetes.pod.uid: affdbe5d0709ec0c8cfe4e796df74130,},Annotations:map[string]string{io.kubernetes.container.hash: d7e5eb98,io.kubernetes.container.restartCount: 5,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:53858589abe0969b0d43fd253c11bdba2cc0c2966139f79c07bbf78cffdcd869,PodSandboxId:7f2d26260bc132df83346e260059f600807bc71e82449c959512f6772a42dc78,Metadata:&ContainerMetadata{Name:kube-scheduler,Attempt:2,},Image:&ImageSpec{Image:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,State:CONTAINER_RUNNING,CreatedAt:1710484272697875290,Labels:map[string]string{io.kubernetes.container.name: kube-scheduler,io.kubernetes.pod.name: kube-scheduler-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernete
s.pod.uid: 06e4fc01158af0abcc722b76c4fcbaee,},Annotations:map[string]string{io.kubernetes.container.hash: e1639c7a,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:e4370ef8479c8071e7f6e97ceb9378aad8d67434c8fb0c47b8d0fc9bda7f3596,PodSandboxId:f7b655acbd70858956764f6cbf8b2cb467f44a747f8c4a0a4fa876282cf4b56a,Metadata:&ContainerMetadata{Name:kube-apiserver,Attempt:4,},Image:&ImageSpec{Image:7fe0e6f37db33464725e616a12ccc4e36970370005a2b09683a974db6350c257,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:7fe0e6f37db33464725e616a12ccc4e36970370005a2b09683a974db6350c257,State:CONTAINER_EXITED,CreatedAt:1710484012956974785,Labels:map[string]string{io.kubernetes.container.name: kube-apiserver,io.kubernetes.pod.name: kube-apiserver-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: ec32969267e
5d443d53332f70d668161,},Annotations:map[string]string{io.kubernetes.container.hash: 4d761a02,io.kubernetes.container.restartCount: 4,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:cb4635f3b41c27cc574f00181fa43463b7015da1cead64c27648cf9f11a76de7,PodSandboxId:b3fef0e73d7bb1217924a1358207053526d7b678fa6c4f039b71d9805717c222,Metadata:&ContainerMetadata{Name:kube-vip,Attempt:4,},Image:&ImageSpec{Image:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,State:CONTAINER_EXITED,CreatedAt:1710483971559144395,Labels:map[string]string{io.kubernetes.container.name: kube-vip,io.kubernetes.pod.name: kube-vip-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: affdbe5d0709ec0c8cfe4e796df74130,},Annotations:map[
string]string{io.kubernetes.container.hash: d7e5eb98,io.kubernetes.container.restartCount: 4,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:d0b44fa63b7f7d4b61990c88d7da99f2d4fb738670e3e6e22354b5a5f3a0c29d,PodSandboxId:344f655b58490a4f777d55dd2c551b578c705e7b4a71a6577daa53c114bbc4a6,Metadata:&ContainerMetadata{Name:busybox,Attempt:1,},Image:&ImageSpec{Image:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,State:CONTAINER_EXITED,CreatedAt:1710483786887352054,Labels:map[string]string{io.kubernetes.container.name: busybox,io.kubernetes.pod.name: busybox-5b5d89c9d6-82knb,io.kubernetes.pod.namespace: default,io.kubernetes.pod.uid: c12d72ab-189f-4a4a-a7df-54e10184a9ac,},Annotations:map[string]string{io.kubernetes.containe
r.hash: 7b2cc69f,io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:a72cadbbcef74cbfe983b247bf865ecdf46f6ca8526deeb1657b9f137651e3f2,PodSandboxId:c28c4fa9bcc72d9300f656805a3e97c6dcdddab3671e12d0480e6e96be9c6391,Metadata:&ContainerMetadata{Name:kube-proxy,Attempt:1,},Image:&ImageSpec{Image:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,State:CONTAINER_EXITED,CreatedAt:1710483753567192605,Labels:map[string]string{io.kubernetes.container.name: kube-proxy,io.kubernetes.pod.name: kube-proxy-sbxgg,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 33fac82d-5f3a-42b8-99b7-1f4ee45c0f98,},Annotations:map[string]string{io.kubernetes.container.hash: 2d71c9ea,io.kubernetes.con
tainer.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:c7b60409e5d22324ba312edc6c4f5e94d3a1b00f5d9f7dc2922756f23a23f31e,PodSandboxId:268a2a5390630a207b2375a2133c6867d83efaacce2f2dcb1508f1a9aaf65799,Metadata:&ContainerMetadata{Name:coredns,Attempt:1,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_EXITED,CreatedAt:1710483753601335090,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-r57px,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 4d7f8905-5519-453d-a9a3-26b5d511f1c3,},Annotations:map[string]string{io.kubernetes.container.hash: 579e5097,io.kubernetes.container.ports: [{\"name\":\"dns\"
,\"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:f1c6cdc2511cf629dab56a84f54e1f95515f47a5fe7d810e843ab351d4c4db1c,PodSandboxId:8899126815da59ab432c51df4ada6cd869277719ab23e03fadd977ebca310687,Metadata:&ContainerMetadata{Name:kube-scheduler,Attempt:1,},Image:&ImageSpec{Image:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,State:CONTAINER_EXITED,CreatedAt:1710483753499514985,Labels:map[string]string{io.kubernetes.container.name: kube-scheduler,io.kubernetes.pod.name: kube-scheduler-ha-866665,io.kubernetes
.pod.namespace: kube-system,io.kubernetes.pod.uid: 06e4fc01158af0abcc722b76c4fcbaee,},Annotations:map[string]string{io.kubernetes.container.hash: e1639c7a,io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:20a813e0950a0ee7b0095bc9f0aeae8c6e0eeac53471a0be98dbeaf701f36b8a,PodSandboxId:79337bac30908f0c1e437fa581563a038ff9e4cfd59fdd43f1d02020f0cac259,Metadata:&ContainerMetadata{Name:etcd,Attempt:1,},Image:&ImageSpec{Image:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,State:CONTAINER_EXITED,CreatedAt:1710483753556829494,Labels:map[string]string{io.kubernetes.container.name: etcd,io.kubernetes.pod.name: etcd-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 7
2b65491f907cd3dd68444500227632e,},Annotations:map[string]string{io.kubernetes.container.hash: ad0ffc58,io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:8507c4a363e13f76515845ccd38eb4fda5ddfbd17bb8eb5a14e138f2607603de,PodSandboxId:a1032069bcb086a4d107a66ed66790e1c54fe03538931ef9a0f95200989338b1,Metadata:&ContainerMetadata{Name:coredns,Attempt:1,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_EXITED,CreatedAt:1710483748541541438,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-mgthb,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 6498160a-372c-4273-a82c-b06c4b7b239b,
},Annotations:map[string]string{io.kubernetes.container.hash: 1b336595,io.kubernetes.container.ports: [{\"name\":\"dns\",\"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},},}" file="otel-collector/interceptors.go:74" id=2df5f215-e10d-48ea-a75d-b294ca76eaaa name=/runtime.v1.RuntimeService/ListContainers
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.506529759Z" level=debug msg="Request: &VersionRequest{Version:,}" file="otel-collector/interceptors.go:62" id=f0d7a535-1450-48ae-b394-e5f7b0e9fe8e name=/runtime.v1.RuntimeService/Version
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.506624792Z" level=debug msg="Response: &VersionResponse{Version:0.1.0,RuntimeName:cri-o,RuntimeVersion:1.29.1,RuntimeApiVersion:v1,}" file="otel-collector/interceptors.go:74" id=f0d7a535-1450-48ae-b394-e5f7b0e9fe8e name=/runtime.v1.RuntimeService/Version
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.509705271Z" level=debug msg="Request: &ImageFsInfoRequest{}" file="otel-collector/interceptors.go:62" id=d1075dcb-bcdb-4e14-a16d-9c32e16ac3d6 name=/runtime.v1.ImageService/ImageFsInfo
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.510159418Z" level=debug msg="Response: &ImageFsInfoResponse{ImageFilesystems:[]*FilesystemUsage{&FilesystemUsage{Timestamp:1710484538510134422,FsId:&FilesystemIdentifier{Mountpoint:/var/lib/containers/storage/overlay-images,},UsedBytes:&UInt64Value{Value:146620,},InodesUsed:&UInt64Value{Value:72,},},},ContainerFilesystems:[]*FilesystemUsage{},}" file="otel-collector/interceptors.go:74" id=d1075dcb-bcdb-4e14-a16d-9c32e16ac3d6 name=/runtime.v1.ImageService/ImageFsInfo
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.510691536Z" level=debug msg="Request: &ListContainersRequest{Filter:&ContainerFilter{Id:,State:nil,PodSandboxId:,LabelSelector:map[string]string{},},}" file="otel-collector/interceptors.go:62" id=af8843bd-7ae0-4611-b068-a7693ca1983a name=/runtime.v1.RuntimeService/ListContainers
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.510782732Z" level=debug msg="No filters were applied, returning full container list" file="server/container_list.go:60" id=af8843bd-7ae0-4611-b068-a7693ca1983a name=/runtime.v1.RuntimeService/ListContainers
Mar 15 06:35:38 ha-866665 crio[6969]: time="2024-03-15 06:35:38.511171935Z" level=debug msg="Response: &ListContainersResponse{Containers:[]*Container{&Container{Id:8f4c9a0644c94c4301a7d96a179f3c1225d5f24d9c926c09380f1e111322dfdd,PodSandboxId:16c817ba7d264b8f7bba3ab1d1ff3074904f7f53258453dc326eaf7fab00203f,Metadata:&ContainerMetadata{Name:kube-controller-manager,Attempt:5,},Image:&ImageSpec{Image:d058aa5ab969ce7b84d25e7188be1f80633b18db8ea7d02d9d0a78e676236591,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:d058aa5ab969ce7b84d25e7188be1f80633b18db8ea7d02d9d0a78e676236591,State:CONTAINER_EXITED,CreatedAt:1710484440561853992,Labels:map[string]string{io.kubernetes.container.name: kube-controller-manager,io.kubernetes.pod.name: kube-controller-manager-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: c0c36bd8aefb52162dc1bcef34f4bb25,},Annotations:map[string]string{io.kubernetes.container.hash: 4b9c51fc,io.kubernetes.container.restartCount: 5,io.kuber
netes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:8568bece5a49f6f0a84fb69937c117029e7dedd90c81050e5a36aef2a90dc985,PodSandboxId:1a1fdba5ec224a16b91ab05b40fb30f9872f97f1e31f57c3bc8180a2d08c9b0e,Metadata:&ContainerMetadata{Name:busybox,Attempt:2,},Image:&ImageSpec{Image:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,State:CONTAINER_RUNNING,CreatedAt:1710484306218979337,Labels:map[string]string{io.kubernetes.container.name: busybox,io.kubernetes.pod.name: busybox-5b5d89c9d6-82knb,io.kubernetes.pod.namespace: default,io.kubernetes.pod.uid: c12d72ab-189f-4a4a-a7df-54e10184a9ac,},Annotations:map[string]string{io.kubernetes.container.hash: 7b2cc69f,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessageP
ath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:10518fb395ccef0d12367b01f3a67613f203a91a536807a5036c49103719fc54,PodSandboxId:d7a69a3a337af71582ce1051a4f71e864ffa1af63897bfe9d000cccced5c310f,Metadata:&ContainerMetadata{Name:kindnet-cni,Attempt:5,},Image:&ImageSpec{Image:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5,State:CONTAINER_EXITED,CreatedAt:1710484278390877828,Labels:map[string]string{io.kubernetes.container.name: kindnet-cni,io.kubernetes.pod.name: kindnet-9nvvx,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 4c5333df-bb98-4f27-9197-875a160f4ff6,},Annotations:map[string]string{io.kubernetes.container.hash: e5c72ddc,io.kubernetes.container.restartCount: 5,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubern
etes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:fd937a8a91decc1f378cd56d4068b095fdceb3673f429ed113e2508c22d3d4a5,PodSandboxId:9f2fb2d671096660329cda96fc359a94e105ee6b24bfcbbfca16d6e253ac9735,Metadata:&ContainerMetadata{Name:kube-proxy,Attempt:2,},Image:&ImageSpec{Image:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,State:CONTAINER_RUNNING,CreatedAt:1710484278257874276,Labels:map[string]string{io.kubernetes.container.name: kube-proxy,io.kubernetes.pod.name: kube-proxy-sbxgg,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 33fac82d-5f3a-42b8-99b7-1f4ee45c0f98,},Annotations:map[string]string{io.kubernetes.container.hash: 2d71c9ea,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessage
Policy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:f31b9e9704e22843a1d46c9994b8fad60de501458b0915ca32d41c10c0f1bde4,PodSandboxId:a0a76006f9e8a838dea8efb6b347aa0d186f4a862312b796d3daefd54f80de2c,Metadata:&ContainerMetadata{Name:storage-provisioner,Attempt:6,},Image:&ImageSpec{Image:6e38f40d628db3002f5617342c8872c935de530d867d0f709a2fbda1a302a562,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:6e38f40d628db3002f5617342c8872c935de530d867d0f709a2fbda1a302a562,State:CONTAINER_EXITED,CreatedAt:1710484278208014606,Labels:map[string]string{io.kubernetes.container.name: storage-provisioner,io.kubernetes.pod.name: storage-provisioner,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: b11128b3-f84e-4526-992d-56e278c3f7c9,},Annotations:map[string]string{io.kubernetes.container.hash: 136548f5,io.kubernetes.container.restartCount: 6,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,
io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:959c3adf756ac670f71e9a23a6963729f6c9b1a1866a5044f8cc579dc7a28be8,PodSandboxId:22072600a839b56ec14de1ab04f264d93721072ea15b5f022bfd1fa67aa1a744,Metadata:&ContainerMetadata{Name:etcd,Attempt:2,},Image:&ImageSpec{Image:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,State:CONTAINER_RUNNING,CreatedAt:1710484278111817672,Labels:map[string]string{io.kubernetes.container.name: etcd,io.kubernetes.pod.name: etcd-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 72b65491f907cd3dd68444500227632e,},Annotations:map[string]string{io.kubernetes.container.hash: ad0ffc58,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&C
ontainer{Id:a3b8244e29a11ef4a5d7f6bc11533ad0fbe55bc470717991ed5d1537f8c04703,PodSandboxId:cf57c2ff9f3b2fd48a06472a048c05a9853e0d41b8f41290305af04765eca0fa,Metadata:&ContainerMetadata{Name:coredns,Attempt:2,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_RUNNING,CreatedAt:1710484272859108005,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-r57px,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 4d7f8905-5519-453d-a9a3-26b5d511f1c3,},Annotations:map[string]string{io.kubernetes.container.hash: 579e5097,io.kubernetes.container.ports: [{\"name\":\"dns\",\"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.contai
ner.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:f8f276ed61ae487693a7e0d2b2f9aa737ceb4003ccb92006a179a894db6abb8a,PodSandboxId:14af170de4b571898874a48dcf002fe94fb21ebca6d8b0ad7fe8f5a1da7e1810,Metadata:&ContainerMetadata{Name:coredns,Attempt:2,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_RUNNING,CreatedAt:1710484272801262973,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-mgthb,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 6498160a-372c-4273-a82c-b06c4b7b239b,},Annotations:map[string]string{io.kubernetes.container.hash: 1b336595,io.kubernetes.container.ports: [{\"name\":\"dns\",\
"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:4632db4347aeb5d329b75f46db342bda137407096006e2ba18e03e6a7ab005a2,PodSandboxId:09c39329a8da2573ad3cbdd895668b0e9a7645b64732383eae88272e9f00894b,Metadata:&ContainerMetadata{Name:kube-vip,Attempt:5,},Image:&ImageSpec{Image:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,State:CONTAINER_RUNNING,CreatedAt:1710484272726972441,Labels:map[string]string{io.kubernetes.container.name: kube-vip,io.kubernetes.pod.name: kube-vip-ha-866665,io.kubernetes.pod.namespace: kub
e-system,io.kubernetes.pod.uid: affdbe5d0709ec0c8cfe4e796df74130,},Annotations:map[string]string{io.kubernetes.container.hash: d7e5eb98,io.kubernetes.container.restartCount: 5,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:53858589abe0969b0d43fd253c11bdba2cc0c2966139f79c07bbf78cffdcd869,PodSandboxId:7f2d26260bc132df83346e260059f600807bc71e82449c959512f6772a42dc78,Metadata:&ContainerMetadata{Name:kube-scheduler,Attempt:2,},Image:&ImageSpec{Image:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,State:CONTAINER_RUNNING,CreatedAt:1710484272697875290,Labels:map[string]string{io.kubernetes.container.name: kube-scheduler,io.kubernetes.pod.name: kube-scheduler-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernete
s.pod.uid: 06e4fc01158af0abcc722b76c4fcbaee,},Annotations:map[string]string{io.kubernetes.container.hash: e1639c7a,io.kubernetes.container.restartCount: 2,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:e4370ef8479c8071e7f6e97ceb9378aad8d67434c8fb0c47b8d0fc9bda7f3596,PodSandboxId:f7b655acbd70858956764f6cbf8b2cb467f44a747f8c4a0a4fa876282cf4b56a,Metadata:&ContainerMetadata{Name:kube-apiserver,Attempt:4,},Image:&ImageSpec{Image:7fe0e6f37db33464725e616a12ccc4e36970370005a2b09683a974db6350c257,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:7fe0e6f37db33464725e616a12ccc4e36970370005a2b09683a974db6350c257,State:CONTAINER_EXITED,CreatedAt:1710484012956974785,Labels:map[string]string{io.kubernetes.container.name: kube-apiserver,io.kubernetes.pod.name: kube-apiserver-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: ec32969267e
5d443d53332f70d668161,},Annotations:map[string]string{io.kubernetes.container.hash: 4d761a02,io.kubernetes.container.restartCount: 4,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:cb4635f3b41c27cc574f00181fa43463b7015da1cead64c27648cf9f11a76de7,PodSandboxId:b3fef0e73d7bb1217924a1358207053526d7b678fa6c4f039b71d9805717c222,Metadata:&ContainerMetadata{Name:kube-vip,Attempt:4,},Image:&ImageSpec{Image:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba,State:CONTAINER_EXITED,CreatedAt:1710483971559144395,Labels:map[string]string{io.kubernetes.container.name: kube-vip,io.kubernetes.pod.name: kube-vip-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: affdbe5d0709ec0c8cfe4e796df74130,},Annotations:map[
string]string{io.kubernetes.container.hash: d7e5eb98,io.kubernetes.container.restartCount: 4,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:d0b44fa63b7f7d4b61990c88d7da99f2d4fb738670e3e6e22354b5a5f3a0c29d,PodSandboxId:344f655b58490a4f777d55dd2c551b578c705e7b4a71a6577daa53c114bbc4a6,Metadata:&ContainerMetadata{Name:busybox,Attempt:1,},Image:&ImageSpec{Image:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a,State:CONTAINER_EXITED,CreatedAt:1710483786887352054,Labels:map[string]string{io.kubernetes.container.name: busybox,io.kubernetes.pod.name: busybox-5b5d89c9d6-82knb,io.kubernetes.pod.namespace: default,io.kubernetes.pod.uid: c12d72ab-189f-4a4a-a7df-54e10184a9ac,},Annotations:map[string]string{io.kubernetes.containe
r.hash: 7b2cc69f,io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:a72cadbbcef74cbfe983b247bf865ecdf46f6ca8526deeb1657b9f137651e3f2,PodSandboxId:c28c4fa9bcc72d9300f656805a3e97c6dcdddab3671e12d0480e6e96be9c6391,Metadata:&ContainerMetadata{Name:kube-proxy,Attempt:1,},Image:&ImageSpec{Image:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e,State:CONTAINER_EXITED,CreatedAt:1710483753567192605,Labels:map[string]string{io.kubernetes.container.name: kube-proxy,io.kubernetes.pod.name: kube-proxy-sbxgg,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 33fac82d-5f3a-42b8-99b7-1f4ee45c0f98,},Annotations:map[string]string{io.kubernetes.container.hash: 2d71c9ea,io.kubernetes.con
tainer.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:c7b60409e5d22324ba312edc6c4f5e94d3a1b00f5d9f7dc2922756f23a23f31e,PodSandboxId:268a2a5390630a207b2375a2133c6867d83efaacce2f2dcb1508f1a9aaf65799,Metadata:&ContainerMetadata{Name:coredns,Attempt:1,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_EXITED,CreatedAt:1710483753601335090,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-r57px,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 4d7f8905-5519-453d-a9a3-26b5d511f1c3,},Annotations:map[string]string{io.kubernetes.container.hash: 579e5097,io.kubernetes.container.ports: [{\"name\":\"dns\"
,\"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:f1c6cdc2511cf629dab56a84f54e1f95515f47a5fe7d810e843ab351d4c4db1c,PodSandboxId:8899126815da59ab432c51df4ada6cd869277719ab23e03fadd977ebca310687,Metadata:&ContainerMetadata{Name:kube-scheduler,Attempt:1,},Image:&ImageSpec{Image:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1,State:CONTAINER_EXITED,CreatedAt:1710483753499514985,Labels:map[string]string{io.kubernetes.container.name: kube-scheduler,io.kubernetes.pod.name: kube-scheduler-ha-866665,io.kubernetes
.pod.namespace: kube-system,io.kubernetes.pod.uid: 06e4fc01158af0abcc722b76c4fcbaee,},Annotations:map[string]string{io.kubernetes.container.hash: e1639c7a,io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:20a813e0950a0ee7b0095bc9f0aeae8c6e0eeac53471a0be98dbeaf701f36b8a,PodSandboxId:79337bac30908f0c1e437fa581563a038ff9e4cfd59fdd43f1d02020f0cac259,Metadata:&ContainerMetadata{Name:etcd,Attempt:1,},Image:&ImageSpec{Image:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9,State:CONTAINER_EXITED,CreatedAt:1710483753556829494,Labels:map[string]string{io.kubernetes.container.name: etcd,io.kubernetes.pod.name: etcd-ha-866665,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 7
2b65491f907cd3dd68444500227632e,},Annotations:map[string]string{io.kubernetes.container.hash: ad0ffc58,io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},&Container{Id:8507c4a363e13f76515845ccd38eb4fda5ddfbd17bb8eb5a14e138f2607603de,PodSandboxId:a1032069bcb086a4d107a66ed66790e1c54fe03538931ef9a0f95200989338b1,Metadata:&ContainerMetadata{Name:coredns,Attempt:1,},Image:&ImageSpec{Image:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,Annotations:map[string]string{},UserSpecifiedImage:,RuntimeHandler:,},ImageRef:ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc,State:CONTAINER_EXITED,CreatedAt:1710483748541541438,Labels:map[string]string{io.kubernetes.container.name: coredns,io.kubernetes.pod.name: coredns-5dd5756b68-mgthb,io.kubernetes.pod.namespace: kube-system,io.kubernetes.pod.uid: 6498160a-372c-4273-a82c-b06c4b7b239b,
},Annotations:map[string]string{io.kubernetes.container.hash: 1b336595,io.kubernetes.container.ports: [{\"name\":\"dns\",\"containerPort\":53,\"protocol\":\"UDP\"},{\"name\":\"dns-tcp\",\"containerPort\":53,\"protocol\":\"TCP\"},{\"name\":\"metrics\",\"containerPort\":9153,\"protocol\":\"TCP\"}],io.kubernetes.container.restartCount: 1,io.kubernetes.container.terminationMessagePath: /dev/termination-log,io.kubernetes.container.terminationMessagePolicy: File,io.kubernetes.pod.terminationGracePeriod: 30,},},},}" file="otel-collector/interceptors.go:74" id=af8843bd-7ae0-4611-b068-a7693ca1983a name=/runtime.v1.RuntimeService/ListContainers
==> container status <==
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
8f4c9a0644c94 d058aa5ab969ce7b84d25e7188be1f80633b18db8ea7d02d9d0a78e676236591 About a minute ago Exited kube-controller-manager 5 16c817ba7d264 kube-controller-manager-ha-866665
8568bece5a49f 8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a 3 minutes ago Running busybox 2 1a1fdba5ec224 busybox-5b5d89c9d6-82knb
10518fb395cce 4950bb10b3f87e8d4a8f772a0d8934625cac4ccfa3675fea34cad0dab83fd5a5 4 minutes ago Exited kindnet-cni 5 d7a69a3a337af kindnet-9nvvx
fd937a8a91dec 83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e 4 minutes ago Running kube-proxy 2 9f2fb2d671096 kube-proxy-sbxgg
f31b9e9704e22 6e38f40d628db3002f5617342c8872c935de530d867d0f709a2fbda1a302a562 4 minutes ago Exited storage-provisioner 6 a0a76006f9e8a storage-provisioner
959c3adf756ac 73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9 4 minutes ago Running etcd 2 22072600a839b etcd-ha-866665
a3b8244e29a11 ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc 4 minutes ago Running coredns 2 cf57c2ff9f3b2 coredns-5dd5756b68-r57px
f8f276ed61ae4 ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc 4 minutes ago Running coredns 2 14af170de4b57 coredns-5dd5756b68-mgthb
4632db4347aeb 22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba 4 minutes ago Running kube-vip 5 09c39329a8da2 kube-vip-ha-866665
53858589abe09 e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1 4 minutes ago Running kube-scheduler 2 7f2d26260bc13 kube-scheduler-ha-866665
e4370ef8479c8 7fe0e6f37db33464725e616a12ccc4e36970370005a2b09683a974db6350c257 8 minutes ago Exited kube-apiserver 4 f7b655acbd708 kube-apiserver-ha-866665
cb4635f3b41c2 22aaebb38f4a9f54562fab7b3a59b206e32f59a368c5749c96d06f5a1c187dba 9 minutes ago Exited kube-vip 4 b3fef0e73d7bb kube-vip-ha-866665
d0b44fa63b7f7 8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a 12 minutes ago Exited busybox 1 344f655b58490 busybox-5b5d89c9d6-82knb
c7b60409e5d22 ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc 13 minutes ago Exited coredns 1 268a2a5390630 coredns-5dd5756b68-r57px
a72cadbbcef74 83f6cc407eed88d214aad97f3539bde5c8e485ff14424cd021a3a2899304398e 13 minutes ago Exited kube-proxy 1 c28c4fa9bcc72 kube-proxy-sbxgg
20a813e0950a0 73deb9a3f702532592a4167455f8bf2e5f5d900bcc959ba2fd2d35c321de1af9 13 minutes ago Exited etcd 1 79337bac30908 etcd-ha-866665
f1c6cdc2511cf e3db313c6dbc065d4ac3b32c7a6f2a878949031b881d217b63881a109c5cfba1 13 minutes ago Exited kube-scheduler 1 8899126815da5 kube-scheduler-ha-866665
8507c4a363e13 ead0a4a53df89fd173874b46093b6e62d8c72967bbf606d672c9e8c9b601a4fc 13 minutes ago Exited coredns 1 a1032069bcb08 coredns-5dd5756b68-mgthb
==> coredns [8507c4a363e13f76515845ccd38eb4fda5ddfbd17bb8eb5a14e138f2607603de] <==
Trace[800224354]: ---"Objects listed" error:Unauthorized 12252ms (06:27:21.915)
Trace[800224354]: [12.252932189s] [12.252932189s] END
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.Namespace: Unauthorized
[INFO] plugin/kubernetes: Trace[532336764]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169 (15-Mar-2024 06:27:10.595) (total time: 11322ms):
Trace[532336764]: ---"Objects listed" error:Unauthorized 11321ms (06:27:21.916)
Trace[532336764]: [11.322096854s] [11.322096854s] END
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.Service: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.Service: failed to list *v1.Service: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.EndpointSlice: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.Namespace: Unauthorized
[INFO] plugin/kubernetes: Trace[1149679676]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169 (15-Mar-2024 06:27:25.734) (total time: 10173ms):
Trace[1149679676]: ---"Objects listed" error:Unauthorized 10171ms (06:27:35.906)
Trace[1149679676]: [10.173827374s] [10.173827374s] END
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?resourceVersion=2450": dial tcp 10.96.0.1:443: connect: connection refused - error from a previous attempt: unexpected EOF
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?resourceVersion=2450": dial tcp 10.96.0.1:443: connect: connection refused - error from a previous attempt: unexpected EOF
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.EndpointSlice: Get "https://10.96.0.1:443/apis/discovery.k8s.io/v1/endpointslices?resourceVersion=2450": dial tcp 10.96.0.1:443: connect: connection refused - error from a previous attempt: unexpected EOF
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://10.96.0.1:443/apis/discovery.k8s.io/v1/endpointslices?resourceVersion=2450": dial tcp 10.96.0.1:443: connect: connection refused - error from a previous attempt: unexpected EOF
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?resourceVersion=2450": dial tcp 10.96.0.1:443: connect: connection refused - error from a previous attempt: unexpected EOF
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?resourceVersion=2450": dial tcp 10.96.0.1:443: connect: connection refused - error from a previous attempt: unexpected EOF
[INFO] SIGTERM: Shutting down servers then terminating
[INFO] plugin/health: Going into lameduck mode for 5s
==> coredns [a3b8244e29a11ef4a5d7f6bc11533ad0fbe55bc470717991ed5d1537f8c04703] <==
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/ready: Still waiting on: "kubernetes"
[WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API
.:53
[INFO] plugin/reload: Running configuration SHA512 = 6c8bd46af3d98e03c4ae8e438c65dd0c69a5f817565481bcf1725dd66ff794963b7938c81e3a23d4c2ad9e52f818076e819219c79e8007dd90564767ed68ba4c
CoreDNS-1.10.1
linux/amd64, go1.20, 055b2c3
[INFO] 127.0.0.1:40115 - 43756 "HINFO IN 2717951387138798829.7180821467390164679. udp 57 false 512" NXDOMAIN qr,rd,ra 57 0.010608616s
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.96.0.1:443/version": dial tcp 10.96.0.1:443: connect: connection refused
[WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.96.0.1:443/version": dial tcp 10.96.0.1:443: connect: connection refused
[INFO] plugin/ready: Still waiting on: "kubernetes"
[WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.96.0.1:443/version": dial tcp 10.96.0.1:443: connect: connection refused
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.96.0.1:443/version": net/http: TLS handshake timeout
==> coredns [c7b60409e5d22324ba312edc6c4f5e94d3a1b00f5d9f7dc2922756f23a23f31e] <==
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.Service: Unauthorized
[INFO] plugin/kubernetes: Trace[652911396]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169 (15-Mar-2024 06:27:24.712) (total time: 11205ms):
Trace[652911396]: ---"Objects listed" error:Unauthorized 11205ms (06:27:35.918)
Trace[652911396]: [11.205453768s] [11.205453768s] END
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.Service: failed to list *v1.Service: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.EndpointSlice: Unauthorized
[INFO] plugin/kubernetes: Trace[659281961]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169 (15-Mar-2024 06:27:24.824) (total time: 11093ms):
Trace[659281961]: ---"Objects listed" error:Unauthorized 11093ms (06:27:35.918)
Trace[659281961]: [11.093964434s] [11.093964434s] END
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.Namespace: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.Service: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.Service: failed to list *v1.Service: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.EndpointSlice: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Unauthorized
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Unauthorized
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?resourceVersion=2450": dial tcp 10.96.0.1:443: connect: connection refused
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.Namespace: failed to list *v1.Namespace: Get "https://10.96.0.1:443/api/v1/namespaces?resourceVersion=2450": dial tcp 10.96.0.1:443: connect: connection refused
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.EndpointSlice: Get "https://10.96.0.1:443/apis/discovery.k8s.io/v1/endpointslices?resourceVersion=2450": dial tcp 10.96.0.1:443: connect: connection refused
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://10.96.0.1:443/apis/discovery.k8s.io/v1/endpointslices?resourceVersion=2450": dial tcp 10.96.0.1:443: connect: connection refused
[INFO] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?resourceVersion=2450": dial tcp 10.96.0.1:443: connect: no route to host
[ERROR] plugin/kubernetes: pkg/mod/k8s.io/client-go@v0.26.1/tools/cache/reflector.go:169: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://10.96.0.1:443/api/v1/services?resourceVersion=2450": dial tcp 10.96.0.1:443: connect: no route to host
[INFO] SIGTERM: Shutting down servers then terminating
[INFO] plugin/health: Going into lameduck mode for 5s
==> coredns [f8f276ed61ae487693a7e0d2b2f9aa737ceb4003ccb92006a179a894db6abb8a] <==
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/ready: Still waiting on: "kubernetes"
[WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API
.:53
[INFO] plugin/reload: Running configuration SHA512 = 6c8bd46af3d98e03c4ae8e438c65dd0c69a5f817565481bcf1725dd66ff794963b7938c81e3a23d4c2ad9e52f818076e819219c79e8007dd90564767ed68ba4c
CoreDNS-1.10.1
linux/amd64, go1.20, 055b2c3
[INFO] 127.0.0.1:47355 - 18813 "HINFO IN 6697230244971142980.3977120107732033871. udp 57 false 512" NXDOMAIN qr,rd,ra 57 0.009991661s
[INFO] plugin/ready: Still waiting on: "kubernetes"
[WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.96.0.1:443/version": dial tcp 10.96.0.1:443: connect: connection refused
[WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.96.0.1:443/version": dial tcp 10.96.0.1:443: connect: connection refused
[INFO] plugin/ready: Still waiting on: "kubernetes"
[WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.96.0.1:443/version": dial tcp 10.96.0.1:443: connect: connection refused
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/ready: Still waiting on: "kubernetes"
[WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.96.0.1:443/version": dial tcp 10.96.0.1:443: connect: connection refused
==> describe nodes <==
command /bin/bash -c "sudo /var/lib/minikube/binaries/v1.28.4/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" failed with error: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.28.4/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:
stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
==> dmesg <==
[ +0.082898] kauditd_printk_skb: 51 callbacks suppressed
[ +12.720187] kauditd_printk_skb: 21 callbacks suppressed
[ +5.465799] kauditd_printk_skb: 38 callbacks suppressed
[Mar15 06:12] kauditd_printk_skb: 26 callbacks suppressed
[Mar15 06:22] systemd-fstab-generator[3836]: Ignoring "noauto" option for root device
[ +0.152680] systemd-fstab-generator[3848]: Ignoring "noauto" option for root device
[ +0.183958] systemd-fstab-generator[3862]: Ignoring "noauto" option for root device
[ +0.159389] systemd-fstab-generator[3874]: Ignoring "noauto" option for root device
[ +0.267565] systemd-fstab-generator[3898]: Ignoring "noauto" option for root device
[ +5.718504] systemd-fstab-generator[4002]: Ignoring "noauto" option for root device
[ +0.088299] kauditd_printk_skb: 100 callbacks suppressed
[ +5.252692] kauditd_printk_skb: 32 callbacks suppressed
[ +21.603346] kauditd_printk_skb: 74 callbacks suppressed
[Mar15 06:23] kauditd_printk_skb: 4 callbacks suppressed
[ +5.000898] kauditd_printk_skb: 4 callbacks suppressed
[Mar15 06:29] systemd-fstab-generator[6878]: Ignoring "noauto" option for root device
[ +0.182963] systemd-fstab-generator[6890]: Ignoring "noauto" option for root device
[ +0.207318] systemd-fstab-generator[6904]: Ignoring "noauto" option for root device
[ +0.160914] systemd-fstab-generator[6916]: Ignoring "noauto" option for root device
[ +0.264473] systemd-fstab-generator[6940]: Ignoring "noauto" option for root device
[Mar15 06:31] systemd-fstab-generator[7068]: Ignoring "noauto" option for root device
[ +0.099499] kauditd_printk_skb: 106 callbacks suppressed
[ +5.728671] kauditd_printk_skb: 62 callbacks suppressed
[ +6.581573] kauditd_printk_skb: 34 callbacks suppressed
[Mar15 06:32] kauditd_printk_skb: 4 callbacks suppressed
==> etcd [20a813e0950a0ee7b0095bc9f0aeae8c6e0eeac53471a0be98dbeaf701f36b8a] <==
{"level":"info","ts":"2024-03-15T06:27:58.935705Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 received MsgPreVoteResp from 83fde65c75733ea3 at term 4"}
{"level":"info","ts":"2024-03-15T06:27:58.93572Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 [logterm: 4, index: 2886] sent MsgPreVote request to af74041eca695613 at term 4"}
{"level":"warn","ts":"2024-03-15T06:27:59.520795Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"af74041eca695613","rtt":"9.002373ms","error":"dial tcp 192.168.39.27:2380: i/o timeout"}
{"level":"warn","ts":"2024-03-15T06:27:59.521075Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"af74041eca695613","rtt":"1.272856ms","error":"dial tcp 192.168.39.27:2380: i/o timeout"}
{"level":"info","ts":"2024-03-15T06:28:00.636Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 is starting a new election at term 4"}
{"level":"info","ts":"2024-03-15T06:28:00.636096Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 became pre-candidate at term 4"}
{"level":"info","ts":"2024-03-15T06:28:00.636112Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 received MsgPreVoteResp from 83fde65c75733ea3 at term 4"}
{"level":"info","ts":"2024-03-15T06:28:00.636133Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 [logterm: 4, index: 2886] sent MsgPreVote request to af74041eca695613 at term 4"}
{"level":"info","ts":"2024-03-15T06:28:00.991925Z","caller":"osutil/interrupt_unix.go:64","msg":"received signal; shutting down","signal":"terminated"}
{"level":"info","ts":"2024-03-15T06:28:00.992032Z","caller":"embed/etcd.go:376","msg":"closing etcd server","name":"ha-866665","data-dir":"/var/lib/minikube/etcd","advertise-peer-urls":["https://192.168.39.78:2380"],"advertise-client-urls":["https://192.168.39.78:2379"]}
{"level":"warn","ts":"2024-03-15T06:28:00.992202Z","caller":"embed/serve.go:212","msg":"stopping secure grpc server due to error","error":"accept tcp 127.0.0.1:2379: use of closed network connection"}
{"level":"warn","ts":"2024-03-15T06:28:00.992285Z","caller":"embed/serve.go:214","msg":"stopped secure grpc server due to error","error":"accept tcp 127.0.0.1:2379: use of closed network connection"}
{"level":"warn","ts":"2024-03-15T06:28:00.994802Z","caller":"embed/serve.go:212","msg":"stopping secure grpc server due to error","error":"accept tcp 192.168.39.78:2379: use of closed network connection"}
{"level":"warn","ts":"2024-03-15T06:28:00.994827Z","caller":"embed/serve.go:214","msg":"stopped secure grpc server due to error","error":"accept tcp 192.168.39.78:2379: use of closed network connection"}
{"level":"info","ts":"2024-03-15T06:28:00.994881Z","caller":"etcdserver/server.go:1456","msg":"skipped leadership transfer; local server is not leader","local-member-id":"83fde65c75733ea3","current-leader-member-id":"0"}
{"level":"info","ts":"2024-03-15T06:28:00.99506Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"af74041eca695613"}
{"level":"info","ts":"2024-03-15T06:28:00.995086Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"af74041eca695613"}
{"level":"info","ts":"2024-03-15T06:28:00.995114Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"af74041eca695613"}
{"level":"info","ts":"2024-03-15T06:28:00.995281Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"83fde65c75733ea3","remote-peer-id":"af74041eca695613"}
{"level":"info","ts":"2024-03-15T06:28:00.995331Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"83fde65c75733ea3","remote-peer-id":"af74041eca695613"}
{"level":"info","ts":"2024-03-15T06:28:00.995423Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"83fde65c75733ea3","remote-peer-id":"af74041eca695613"}
{"level":"info","ts":"2024-03-15T06:28:00.995436Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"af74041eca695613"}
{"level":"info","ts":"2024-03-15T06:28:01.013373Z","caller":"embed/etcd.go:579","msg":"stopping serving peer traffic","address":"192.168.39.78:2380"}
{"level":"info","ts":"2024-03-15T06:28:01.013771Z","caller":"embed/etcd.go:584","msg":"stopped serving peer traffic","address":"192.168.39.78:2380"}
{"level":"info","ts":"2024-03-15T06:28:01.01388Z","caller":"embed/etcd.go:378","msg":"closed etcd server","name":"ha-866665","data-dir":"/var/lib/minikube/etcd","advertise-peer-urls":["https://192.168.39.78:2380"],"advertise-client-urls":["https://192.168.39.78:2379"]}
==> etcd [959c3adf756ac670f71e9a23a6963729f6c9b1a1866a5044f8cc579dc7a28be8] <==
{"level":"info","ts":"2024-03-15T06:31:59.650477Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"83fde65c75733ea3","remote-peer-id":"af74041eca695613"}
{"level":"info","ts":"2024-03-15T06:31:59.664413Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"83fde65c75733ea3","to":"af74041eca695613","stream-type":"stream MsgApp v2"}
{"level":"info","ts":"2024-03-15T06:31:59.664504Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"83fde65c75733ea3","remote-peer-id":"af74041eca695613"}
{"level":"info","ts":"2024-03-15T06:31:59.6685Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"83fde65c75733ea3","to":"af74041eca695613","stream-type":"stream Message"}
{"level":"info","ts":"2024-03-15T06:31:59.668598Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"83fde65c75733ea3","remote-peer-id":"af74041eca695613"}
{"level":"info","ts":"2024-03-15T06:32:00.258901Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 is starting a new election at term 4"}
{"level":"info","ts":"2024-03-15T06:32:00.258942Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 became pre-candidate at term 4"}
{"level":"info","ts":"2024-03-15T06:32:00.258953Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 received MsgPreVoteResp from 83fde65c75733ea3 at term 4"}
{"level":"info","ts":"2024-03-15T06:32:00.258974Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 [logterm: 4, index: 2886] sent MsgPreVote request to af74041eca695613 at term 4"}
{"level":"info","ts":"2024-03-15T06:32:00.260102Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 received MsgPreVoteResp from af74041eca695613 at term 4"}
{"level":"info","ts":"2024-03-15T06:32:00.260177Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 has received 2 MsgPreVoteResp votes and 0 vote rejections"}
{"level":"info","ts":"2024-03-15T06:32:00.260314Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 became candidate at term 5"}
{"level":"info","ts":"2024-03-15T06:32:00.260345Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 received MsgVoteResp from 83fde65c75733ea3 at term 5"}
{"level":"info","ts":"2024-03-15T06:32:00.260375Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 [logterm: 4, index: 2886] sent MsgVote request to af74041eca695613 at term 5"}
{"level":"info","ts":"2024-03-15T06:32:00.266101Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 received MsgVoteResp from af74041eca695613 at term 5"}
{"level":"info","ts":"2024-03-15T06:32:00.266153Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 has received 2 MsgVoteResp votes and 0 vote rejections"}
{"level":"info","ts":"2024-03-15T06:32:00.266182Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"83fde65c75733ea3 became leader at term 5"}
{"level":"info","ts":"2024-03-15T06:32:00.266194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 83fde65c75733ea3 elected leader 83fde65c75733ea3 at term 5"}
{"level":"info","ts":"2024-03-15T06:32:00.273166Z","caller":"etcdserver/server.go:2062","msg":"published local member to cluster through raft","local-member-id":"83fde65c75733ea3","local-member-attributes":"{Name:ha-866665 ClientURLs:[https://192.168.39.78:2379]}","request-path":"/0/members/83fde65c75733ea3/attributes","cluster-id":"254f9db842b1870b","publish-timeout":"7s"}
{"level":"info","ts":"2024-03-15T06:32:00.273534Z","caller":"embed/serve.go:103","msg":"ready to serve client requests"}
{"level":"info","ts":"2024-03-15T06:32:00.275869Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc+http","address":"127.0.0.1:2379"}
{"level":"info","ts":"2024-03-15T06:32:00.275964Z","caller":"embed/serve.go:103","msg":"ready to serve client requests"}
{"level":"info","ts":"2024-03-15T06:32:00.277324Z","caller":"embed/serve.go:250","msg":"serving client traffic securely","traffic":"grpc+http","address":"192.168.39.78:2379"}
{"level":"info","ts":"2024-03-15T06:32:00.277644Z","caller":"etcdmain/main.go:44","msg":"notifying init daemon"}
{"level":"info","ts":"2024-03-15T06:32:00.277708Z","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"}
==> kernel <==
06:35:39 up 25 min, 0 users, load average: 0.10, 0.19, 0.27
Linux ha-866665 5.10.207 #1 SMP Fri Mar 15 04:22:03 UTC 2024 x86_64 GNU/Linux
PRETTY_NAME="Buildroot 2023.02.9"
==> kindnet [10518fb395ccef0d12367b01f3a67613f203a91a536807a5036c49103719fc54] <==
I0315 06:31:18.825608 1 main.go:150] noMask IPv4 subnets: [10.244.0.0/16]
I0315 06:31:19.125113 1 main.go:191] Failed to get nodes, retrying after error: Get "https://10.96.0.1:443/api/v1/nodes": dial tcp 10.96.0.1:443: connect: connection refused
I0315 06:31:19.440688 1 main.go:191] Failed to get nodes, retrying after error: Get "https://10.96.0.1:443/api/v1/nodes": dial tcp 10.96.0.1:443: connect: no route to host
I0315 06:31:22.512472 1 main.go:191] Failed to get nodes, retrying after error: Get "https://10.96.0.1:443/api/v1/nodes": dial tcp 10.96.0.1:443: connect: no route to host
I0315 06:31:24.514025 1 main.go:191] Failed to get nodes, retrying after error: Get "https://10.96.0.1:443/api/v1/nodes": dial tcp 10.96.0.1:443: connect: connection refused
I0315 06:31:27.514923 1 main.go:191] Failed to get nodes, retrying after error: Get "https://10.96.0.1:443/api/v1/nodes": dial tcp 10.96.0.1:443: connect: connection refused
panic: Reached maximum retries obtaining node list: Get "https://10.96.0.1:443/api/v1/nodes": dial tcp 10.96.0.1:443: connect: connection refused
goroutine 1 [running]:
main.main()
/go/src/cmd/kindnetd/main.go:195 +0xd3d
==> kube-apiserver [e4370ef8479c8071e7f6e97ceb9378aad8d67434c8fb0c47b8d0fc9bda7f3596] <==
W0315 06:27:50.003349 1 reflector.go:535] storage/cacher.go:/volumeattachments: failed to list *storage.VolumeAttachment: etcdserver: request timed out
I0315 06:27:50.003365 1 trace.go:236] Trace[731904531]: "Reflector ListAndWatch" name:storage/cacher.go:/volumeattachments (15-Mar-2024 06:27:36.912) (total time: 13090ms):
Trace[731904531]: ---"Objects listed" error:etcdserver: request timed out 13090ms (06:27:50.003)
Trace[731904531]: [13.090756124s] [13.090756124s] END
E0315 06:27:50.003369 1 cacher.go:470] cacher (volumeattachments.storage.k8s.io): unexpected ListAndWatch error: failed to list *storage.VolumeAttachment: etcdserver: request timed out; reinitializing...
W0315 06:27:50.003380 1 reflector.go:535] storage/cacher.go:/prioritylevelconfigurations: failed to list *flowcontrol.PriorityLevelConfiguration: etcdserver: request timed out
I0315 06:27:50.003393 1 trace.go:236] Trace[592093521]: "Reflector ListAndWatch" name:storage/cacher.go:/prioritylevelconfigurations (15-Mar-2024 06:27:36.904) (total time: 13099ms):
Trace[592093521]: ---"Objects listed" error:etcdserver: request timed out 13099ms (06:27:50.003)
Trace[592093521]: [13.09928422s] [13.09928422s] END
E0315 06:27:50.003397 1 cacher.go:470] cacher (prioritylevelconfigurations.flowcontrol.apiserver.k8s.io): unexpected ListAndWatch error: failed to list *flowcontrol.PriorityLevelConfiguration: etcdserver: request timed out; reinitializing...
W0315 06:27:50.003439 1 reflector.go:535] storage/cacher.go:/poddisruptionbudgets: failed to list *policy.PodDisruptionBudget: etcdserver: request timed out
I0315 06:27:50.003456 1 trace.go:236] Trace[746771974]: "Reflector ListAndWatch" name:storage/cacher.go:/poddisruptionbudgets (15-Mar-2024 06:27:36.904) (total time: 13099ms):
Trace[746771974]: ---"Objects listed" error:etcdserver: request timed out 13099ms (06:27:50.003)
Trace[746771974]: [13.099292505s] [13.099292505s] END
E0315 06:27:50.003482 1 cacher.go:470] cacher (poddisruptionbudgets.policy): unexpected ListAndWatch error: failed to list *policy.PodDisruptionBudget: etcdserver: request timed out; reinitializing...
W0315 06:27:50.003501 1 reflector.go:535] storage/cacher.go:/flowschemas: failed to list *flowcontrol.FlowSchema: etcdserver: request timed out
I0315 06:27:50.003534 1 trace.go:236] Trace[1529918640]: "Reflector ListAndWatch" name:storage/cacher.go:/flowschemas (15-Mar-2024 06:27:36.900) (total time: 13103ms):
Trace[1529918640]: ---"Objects listed" error:etcdserver: request timed out 13103ms (06:27:50.003)
Trace[1529918640]: [13.10350995s] [13.10350995s] END
E0315 06:27:50.003539 1 cacher.go:470] cacher (flowschemas.flowcontrol.apiserver.k8s.io): unexpected ListAndWatch error: failed to list *flowcontrol.FlowSchema: etcdserver: request timed out; reinitializing...
W0315 06:27:50.003551 1 reflector.go:535] storage/cacher.go:/serviceaccounts: failed to list *core.ServiceAccount: etcdserver: request timed out
I0315 06:27:50.003567 1 trace.go:236] Trace[1142995160]: "Reflector ListAndWatch" name:storage/cacher.go:/serviceaccounts (15-Mar-2024 06:27:36.904) (total time: 13099ms):
Trace[1142995160]: ---"Objects listed" error:etcdserver: request timed out 13099ms (06:27:50.003)
Trace[1142995160]: [13.099504673s] [13.099504673s] END
E0315 06:27:50.003590 1 cacher.go:470] cacher (serviceaccounts): unexpected ListAndWatch error: failed to list *core.ServiceAccount: etcdserver: request timed out; reinitializing...
==> kube-controller-manager [8f4c9a0644c94c4301a7d96a179f3c1225d5f24d9c926c09380f1e111322dfdd] <==
I0315 06:34:01.115770 1 serving.go:348] Generated self-signed cert in-memory
I0315 06:34:02.015947 1 controllermanager.go:189] "Starting" version="v1.28.4"
I0315 06:34:02.015993 1 controllermanager.go:191] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0315 06:34:02.025496 1 secure_serving.go:213] Serving securely on 127.0.0.1:10257
I0315 06:34:02.025650 1 dynamic_cafile_content.go:157] "Starting controller" name="request-header::/var/lib/minikube/certs/front-proxy-ca.crt"
I0315 06:34:02.026071 1 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/var/lib/minikube/certs/ca.crt"
I0315 06:34:02.026580 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
E0315 06:34:12.027833 1 controllermanager.go:235] "Error building controller context" err="failed to wait for apiserver being healthy: timed out waiting for the condition: failed to get apiserver /healthz status: Get \"https://192.168.39.78:8443/healthz\": dial tcp 192.168.39.78:8443: connect: connection refused"
==> kube-proxy [a72cadbbcef74cbfe983b247bf865ecdf46f6ca8526deeb1657b9f137651e3f2] <==
E0315 06:26:01.167720 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://control-plane.minikube.internal:8443/api/v1/nodes?fieldSelector=metadata.name%!D(MISSING)ha-866665&resourceVersion=2239": dial tcp 192.168.39.254:8443: connect: no route to host
W0315 06:26:01.167669 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.EndpointSlice: Get "https://control-plane.minikube.internal:8443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2435": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:26:01.167846 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://control-plane.minikube.internal:8443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2435": dial tcp 192.168.39.254:8443: connect: no route to host
W0315 06:26:04.241012 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Service: Get "https://control-plane.minikube.internal:8443/api/v1/services?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2267": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:26:04.241169 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://control-plane.minikube.internal:8443/api/v1/services?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2267": dial tcp 192.168.39.254:8443: connect: no route to host
W0315 06:26:13.456306 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.EndpointSlice: Get "https://control-plane.minikube.internal:8443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2435": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:26:13.456506 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://control-plane.minikube.internal:8443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2435": dial tcp 192.168.39.254:8443: connect: no route to host
W0315 06:26:13.457153 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Node: Get "https://control-plane.minikube.internal:8443/api/v1/nodes?fieldSelector=metadata.name%!D(MISSING)ha-866665&resourceVersion=2239": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:26:13.457208 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://control-plane.minikube.internal:8443/api/v1/nodes?fieldSelector=metadata.name%!D(MISSING)ha-866665&resourceVersion=2239": dial tcp 192.168.39.254:8443: connect: no route to host
W0315 06:26:16.532382 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Service: Get "https://control-plane.minikube.internal:8443/api/v1/services?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2267": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:26:16.532484 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://control-plane.minikube.internal:8443/api/v1/services?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2267": dial tcp 192.168.39.254:8443: connect: no route to host
W0315 06:26:34.959861 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.EndpointSlice: Get "https://control-plane.minikube.internal:8443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2435": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:26:34.959939 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://control-plane.minikube.internal:8443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2435": dial tcp 192.168.39.254:8443: connect: no route to host
W0315 06:26:38.032403 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Node: Get "https://control-plane.minikube.internal:8443/api/v1/nodes?fieldSelector=metadata.name%!D(MISSING)ha-866665&resourceVersion=2239": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:26:38.032468 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://control-plane.minikube.internal:8443/api/v1/nodes?fieldSelector=metadata.name%!D(MISSING)ha-866665&resourceVersion=2239": dial tcp 192.168.39.254:8443: connect: no route to host
W0315 06:26:38.032844 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Service: Get "https://control-plane.minikube.internal:8443/api/v1/services?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2267": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:26:38.032946 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://control-plane.minikube.internal:8443/api/v1/services?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2267": dial tcp 192.168.39.254:8443: connect: no route to host
W0315 06:27:14.897385 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Service: Get "https://control-plane.minikube.internal:8443/api/v1/services?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2267": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:27:14.897592 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://control-plane.minikube.internal:8443/api/v1/services?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2267": dial tcp 192.168.39.254:8443: connect: no route to host
W0315 06:27:17.967871 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Node: Get "https://control-plane.minikube.internal:8443/api/v1/nodes?fieldSelector=metadata.name%!D(MISSING)ha-866665&resourceVersion=2239": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:27:17.968263 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://control-plane.minikube.internal:8443/api/v1/nodes?fieldSelector=metadata.name%!D(MISSING)ha-866665&resourceVersion=2239": dial tcp 192.168.39.254:8443: connect: no route to host
W0315 06:27:21.039682 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.EndpointSlice: Get "https://control-plane.minikube.internal:8443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2435": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:27:21.039794 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://control-plane.minikube.internal:8443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&resourceVersion=2435": dial tcp 192.168.39.254:8443: connect: no route to host
W0315 06:27:54.832570 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Node: Get "https://control-plane.minikube.internal:8443/api/v1/nodes?fieldSelector=metadata.name%!D(MISSING)ha-866665&resourceVersion=2239": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:27:54.832649 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://control-plane.minikube.internal:8443/api/v1/nodes?fieldSelector=metadata.name%!D(MISSING)ha-866665&resourceVersion=2239": dial tcp 192.168.39.254:8443: connect: no route to host
==> kube-proxy [fd937a8a91decc1f378cd56d4068b095fdceb3673f429ed113e2508c22d3d4a5] <==
E0315 06:31:42.162745 1 node.go:130] Failed to retrieve node info: Get "https://control-plane.minikube.internal:8443/api/v1/nodes/ha-866665": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:32:03.665169 1 node.go:130] Failed to retrieve node info: Get "https://control-plane.minikube.internal:8443/api/v1/nodes/ha-866665": dial tcp 192.168.39.254:8443: connect: no route to host
I0315 06:32:03.665837 1 server.go:969] "Can't determine this node's IP, assuming 127.0.0.1; if this is incorrect, please set the --bind-address flag"
I0315 06:32:03.704165 1 server_others.go:121] "No iptables support for family" ipFamily="IPv6"
I0315 06:32:03.704332 1 server.go:634] "kube-proxy running in single-stack mode" ipFamily="IPv4"
I0315 06:32:03.707769 1 server_others.go:152] "Using iptables Proxier"
I0315 06:32:03.707920 1 proxier.go:251] "Setting route_localnet=1 to allow node-ports on localhost; to change this either disable iptables.localhostNodePorts (--iptables-localhost-nodeports) or set nodePortAddresses (--nodeport-addresses) to filter loopback addresses"
I0315 06:32:03.709020 1 server.go:846] "Version info" version="v1.28.4"
I0315 06:32:03.709101 1 server.go:848] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0315 06:32:03.711683 1 config.go:188] "Starting service config controller"
I0315 06:32:03.711754 1 shared_informer.go:311] Waiting for caches to sync for service config
I0315 06:32:03.711789 1 config.go:97] "Starting endpoint slice config controller"
I0315 06:32:03.711825 1 shared_informer.go:311] Waiting for caches to sync for endpoint slice config
I0315 06:32:03.712712 1 config.go:315] "Starting node config controller"
I0315 06:32:03.712754 1 shared_informer.go:311] Waiting for caches to sync for node config
E0315 06:32:06.736537 1 event_broadcaster.go:274] Unable to write event: 'Post "https://control-plane.minikube.internal:8443/apis/events.k8s.io/v1/namespaces/default/events": dial tcp 192.168.39.254:8443: connect: no route to host' (may retry after sleeping)
W0315 06:32:06.736941 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Node: Get "https://control-plane.minikube.internal:8443/api/v1/nodes?fieldSelector=metadata.name%!D(MISSING)ha-866665&limit=500&resourceVersion=0": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:32:06.737313 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://control-plane.minikube.internal:8443/api/v1/nodes?fieldSelector=metadata.name%!D(MISSING)ha-866665&limit=500&resourceVersion=0": dial tcp 192.168.39.254:8443: connect: no route to host
W0315 06:32:06.737542 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Service: Get "https://control-plane.minikube.internal:8443/api/v1/services?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&limit=500&resourceVersion=0": dial tcp 192.168.39.254:8443: connect: no route to host
W0315 06:32:06.737612 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.EndpointSlice: Get "https://control-plane.minikube.internal:8443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&limit=500&resourceVersion=0": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:32:06.737802 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.EndpointSlice: failed to list *v1.EndpointSlice: Get "https://control-plane.minikube.internal:8443/apis/discovery.k8s.io/v1/endpointslices?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&limit=500&resourceVersion=0": dial tcp 192.168.39.254:8443: connect: no route to host
E0315 06:32:06.737668 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://control-plane.minikube.internal:8443/api/v1/services?labelSelector=%!s(MISSING)ervice.kubernetes.io%!F(MISSING)headless%!C(MISSING)%!s(MISSING)ervice.kubernetes.io%!F(MISSING)service-proxy-name&limit=500&resourceVersion=0": dial tcp 192.168.39.254:8443: connect: no route to host
I0315 06:32:07.612784 1 shared_informer.go:318] Caches are synced for service config
I0315 06:32:08.012336 1 shared_informer.go:318] Caches are synced for endpoint slice config
I0315 06:32:08.013310 1 shared_informer.go:318] Caches are synced for node config
==> kube-scheduler [53858589abe0969b0d43fd253c11bdba2cc0c2966139f79c07bbf78cffdcd869] <==
E0315 06:35:02.433064 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: Get "https://192.168.39.78:8443/apis/apps/v1/replicasets?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
W0315 06:35:04.291643 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.CSINode: Get "https://192.168.39.78:8443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
E0315 06:35:04.291746 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.CSINode: failed to list *v1.CSINode: Get "https://192.168.39.78:8443/apis/storage.k8s.io/v1/csinodes?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
W0315 06:35:08.964028 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.PersistentVolumeClaim: Get "https://192.168.39.78:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
E0315 06:35:08.964097 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: Get "https://192.168.39.78:8443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
W0315 06:35:14.007535 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.StatefulSet: Get "https://192.168.39.78:8443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
E0315 06:35:14.007679 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: Get "https://192.168.39.78:8443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
W0315 06:35:16.430156 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.CSIStorageCapacity: Get "https://192.168.39.78:8443/apis/storage.k8s.io/v1/csistoragecapacities?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
E0315 06:35:16.430457 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.CSIStorageCapacity: failed to list *v1.CSIStorageCapacity: Get "https://192.168.39.78:8443/apis/storage.k8s.io/v1/csistoragecapacities?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
W0315 06:35:18.622486 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.PersistentVolume: Get "https://192.168.39.78:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
E0315 06:35:18.622559 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: Get "https://192.168.39.78:8443/api/v1/persistentvolumes?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
W0315 06:35:23.506725 1 reflector.go:535] pkg/server/dynamiccertificates/configmap_cafile_content.go:206: failed to list *v1.ConfigMap: Get "https://192.168.39.78:8443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%!D(MISSING)extension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
E0315 06:35:23.506825 1 reflector.go:147] pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: Get "https://192.168.39.78:8443/api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%!D(MISSING)extension-apiserver-authentication&limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
W0315 06:35:23.886417 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.StorageClass: Get "https://192.168.39.78:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
E0315 06:35:23.886503 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get "https://192.168.39.78:8443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
W0315 06:35:28.295205 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.CSIDriver: Get "https://192.168.39.78:8443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
E0315 06:35:28.295388 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://192.168.39.78:8443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
W0315 06:35:31.864716 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.ReplicationController: Get "https://192.168.39.78:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
E0315 06:35:31.864806 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: Get "https://192.168.39.78:8443/api/v1/replicationcontrollers?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
W0315 06:35:35.412598 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.PodDisruptionBudget: Get "https://192.168.39.78:8443/apis/policy/v1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
E0315 06:35:35.412708 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: Get "https://192.168.39.78:8443/apis/policy/v1/poddisruptionbudgets?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
W0315 06:35:37.486053 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Service: Get "https://192.168.39.78:8443/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
E0315 06:35:37.486164 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://192.168.39.78:8443/api/v1/services?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
W0315 06:35:38.745098 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Node: Get "https://192.168.39.78:8443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
E0315 06:35:38.745152 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://192.168.39.78:8443/api/v1/nodes?limit=500&resourceVersion=0": dial tcp 192.168.39.78:8443: connect: connection refused
==> kube-scheduler [f1c6cdc2511cf629dab56a84f54e1f95515f47a5fe7d810e843ab351d4c4db1c] <==
W0315 06:27:33.721173 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0315 06:27:33.721276 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
W0315 06:27:34.581491 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.CSIStorageCapacity: csistoragecapacities.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csistoragecapacities" in API group "storage.k8s.io" at the cluster scope
E0315 06:27:34.581544 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.CSIStorageCapacity: failed to list *v1.CSIStorageCapacity: csistoragecapacities.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csistoragecapacities" in API group "storage.k8s.io" at the cluster scope
W0315 06:27:36.411769 1 reflector.go:535] pkg/server/dynamiccertificates/configmap_cafile_content.go:206: failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
E0315 06:27:36.411881 1 reflector.go:147] pkg/server/dynamiccertificates/configmap_cafile_content.go:206: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
W0315 06:27:36.473470 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0315 06:27:36.473532 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
W0315 06:27:37.175018 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0315 06:27:37.175090 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
W0315 06:27:38.621446 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0315 06:27:38.621559 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
W0315 06:27:39.985765 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0315 06:27:39.985857 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.CSINode: failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
W0315 06:27:41.948412 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csidrivers" in API group "storage.k8s.io" at the cluster scope
E0315 06:27:41.948471 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csidrivers" in API group "storage.k8s.io" at the cluster scope
W0315 06:27:58.053579 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.ReplicaSet: Get "https://192.168.39.78:8443/apis/apps/v1/replicasets?resourceVersion=2450": dial tcp 192.168.39.78:8443: connect: connection refused
E0315 06:27:58.053849 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: Get "https://192.168.39.78:8443/apis/apps/v1/replicasets?resourceVersion=2450": dial tcp 192.168.39.78:8443: connect: connection refused
W0315 06:27:58.945885 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Pod: Get "https://192.168.39.78:8443/api/v1/pods?fieldSelector=status.phase%3DSucceeded%!C(MISSING)status.phase%3DFailed&resourceVersion=2450": dial tcp 192.168.39.78:8443: connect: connection refused
E0315 06:27:58.945942 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Pod: failed to list *v1.Pod: Get "https://192.168.39.78:8443/api/v1/pods?fieldSelector=status.phase%3DSucceeded%!C(MISSING)status.phase%3DFailed&resourceVersion=2450": dial tcp 192.168.39.78:8443: connect: connection refused
W0315 06:28:00.884506 1 reflector.go:535] vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.StorageClass: Get "https://192.168.39.78:8443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=2450": dial tcp 192.168.39.78:8443: connect: connection refused
E0315 06:28:00.884572 1 reflector.go:147] vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: Get "https://192.168.39.78:8443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=2450": dial tcp 192.168.39.78:8443: connect: connection refused
I0315 06:28:00.994442 1 secure_serving.go:258] Stopped listening on 127.0.0.1:10259
I0315 06:28:00.994493 1 tlsconfig.go:255] "Shutting down DynamicServingCertificateController"
E0315 06:28:00.994655 1 run.go:74] "command failed" err="finished without leader elect"
==> kubelet <==
Mar 15 06:35:05 ha-866665 kubelet[1369]: > table="nat" chain="KUBE-KUBELET-CANARY"
Mar 15 06:35:10 ha-866665 kubelet[1369]: I0315 06:35:10.545742 1369 scope.go:117] "RemoveContainer" containerID="f31b9e9704e22843a1d46c9994b8fad60de501458b0915ca32d41c10c0f1bde4"
Mar 15 06:35:10 ha-866665 kubelet[1369]: E0315 06:35:10.546498 1369 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-provisioner\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=storage-provisioner pod=storage-provisioner_kube-system(b11128b3-f84e-4526-992d-56e278c3f7c9)\"" pod="kube-system/storage-provisioner" podUID="b11128b3-f84e-4526-992d-56e278c3f7c9"
Mar 15 06:35:10 ha-866665 kubelet[1369]: E0315 06:35:10.549941 1369 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = pod sandbox with name \"k8s_kube-apiserver-ha-866665_kube-system_ec32969267e5d443d53332f70d668161_2\" already exists"
Mar 15 06:35:10 ha-866665 kubelet[1369]: E0315 06:35:10.549982 1369 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = pod sandbox with name \"k8s_kube-apiserver-ha-866665_kube-system_ec32969267e5d443d53332f70d668161_2\" already exists" pod="kube-system/kube-apiserver-ha-866665"
Mar 15 06:35:10 ha-866665 kubelet[1369]: E0315 06:35:10.550024 1369 kuberuntime_manager.go:1171] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = pod sandbox with name \"k8s_kube-apiserver-ha-866665_kube-system_ec32969267e5d443d53332f70d668161_2\" already exists" pod="kube-system/kube-apiserver-ha-866665"
Mar 15 06:35:10 ha-866665 kubelet[1369]: E0315 06:35:10.550095 1369 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-apiserver-ha-866665_kube-system(ec32969267e5d443d53332f70d668161)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-apiserver-ha-866665_kube-system(ec32969267e5d443d53332f70d668161)\\\": rpc error: code = Unknown desc = pod sandbox with name \\\"k8s_kube-apiserver-ha-866665_kube-system_ec32969267e5d443d53332f70d668161_2\\\" already exists\"" pod="kube-system/kube-apiserver-ha-866665" podUID="ec32969267e5d443d53332f70d668161"
Mar 15 06:35:13 ha-866665 kubelet[1369]: I0315 06:35:13.544853 1369 scope.go:117] "RemoveContainer" containerID="10518fb395ccef0d12367b01f3a67613f203a91a536807a5036c49103719fc54"
Mar 15 06:35:13 ha-866665 kubelet[1369]: E0315 06:35:13.545764 1369 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kindnet-cni\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=kindnet-cni pod=kindnet-9nvvx_kube-system(4c5333df-bb98-4f27-9197-875a160f4ff6)\"" pod="kube-system/kindnet-9nvvx" podUID="4c5333df-bb98-4f27-9197-875a160f4ff6"
Mar 15 06:35:15 ha-866665 kubelet[1369]: I0315 06:35:15.544672 1369 scope.go:117] "RemoveContainer" containerID="8f4c9a0644c94c4301a7d96a179f3c1225d5f24d9c926c09380f1e111322dfdd"
Mar 15 06:35:15 ha-866665 kubelet[1369]: E0315 06:35:15.545381 1369 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-controller-manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=kube-controller-manager pod=kube-controller-manager-ha-866665_kube-system(c0c36bd8aefb52162dc1bcef34f4bb25)\"" pod="kube-system/kube-controller-manager-ha-866665" podUID="c0c36bd8aefb52162dc1bcef34f4bb25"
Mar 15 06:35:24 ha-866665 kubelet[1369]: I0315 06:35:24.544088 1369 scope.go:117] "RemoveContainer" containerID="10518fb395ccef0d12367b01f3a67613f203a91a536807a5036c49103719fc54"
Mar 15 06:35:24 ha-866665 kubelet[1369]: E0315 06:35:24.544466 1369 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kindnet-cni\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=kindnet-cni pod=kindnet-9nvvx_kube-system(4c5333df-bb98-4f27-9197-875a160f4ff6)\"" pod="kube-system/kindnet-9nvvx" podUID="4c5333df-bb98-4f27-9197-875a160f4ff6"
Mar 15 06:35:24 ha-866665 kubelet[1369]: I0315 06:35:24.544560 1369 scope.go:117] "RemoveContainer" containerID="f31b9e9704e22843a1d46c9994b8fad60de501458b0915ca32d41c10c0f1bde4"
Mar 15 06:35:24 ha-866665 kubelet[1369]: E0315 06:35:24.544801 1369 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-provisioner\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=storage-provisioner pod=storage-provisioner_kube-system(b11128b3-f84e-4526-992d-56e278c3f7c9)\"" pod="kube-system/storage-provisioner" podUID="b11128b3-f84e-4526-992d-56e278c3f7c9"
Mar 15 06:35:25 ha-866665 kubelet[1369]: E0315 06:35:25.550967 1369 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = pod sandbox with name \"k8s_kube-apiserver-ha-866665_kube-system_ec32969267e5d443d53332f70d668161_2\" already exists"
Mar 15 06:35:25 ha-866665 kubelet[1369]: E0315 06:35:25.551026 1369 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = pod sandbox with name \"k8s_kube-apiserver-ha-866665_kube-system_ec32969267e5d443d53332f70d668161_2\" already exists" pod="kube-system/kube-apiserver-ha-866665"
Mar 15 06:35:25 ha-866665 kubelet[1369]: E0315 06:35:25.551048 1369 kuberuntime_manager.go:1171] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = pod sandbox with name \"k8s_kube-apiserver-ha-866665_kube-system_ec32969267e5d443d53332f70d668161_2\" already exists" pod="kube-system/kube-apiserver-ha-866665"
Mar 15 06:35:25 ha-866665 kubelet[1369]: E0315 06:35:25.551114 1369 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-apiserver-ha-866665_kube-system(ec32969267e5d443d53332f70d668161)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-apiserver-ha-866665_kube-system(ec32969267e5d443d53332f70d668161)\\\": rpc error: code = Unknown desc = pod sandbox with name \\\"k8s_kube-apiserver-ha-866665_kube-system_ec32969267e5d443d53332f70d668161_2\\\" already exists\"" pod="kube-system/kube-apiserver-ha-866665" podUID="ec32969267e5d443d53332f70d668161"
Mar 15 06:35:29 ha-866665 kubelet[1369]: I0315 06:35:29.544575 1369 scope.go:117] "RemoveContainer" containerID="8f4c9a0644c94c4301a7d96a179f3c1225d5f24d9c926c09380f1e111322dfdd"
Mar 15 06:35:29 ha-866665 kubelet[1369]: E0315 06:35:29.544964 1369 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kube-controller-manager\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=kube-controller-manager pod=kube-controller-manager-ha-866665_kube-system(c0c36bd8aefb52162dc1bcef34f4bb25)\"" pod="kube-system/kube-controller-manager-ha-866665" podUID="c0c36bd8aefb52162dc1bcef34f4bb25"
Mar 15 06:35:38 ha-866665 kubelet[1369]: I0315 06:35:38.544547 1369 scope.go:117] "RemoveContainer" containerID="f31b9e9704e22843a1d46c9994b8fad60de501458b0915ca32d41c10c0f1bde4"
Mar 15 06:35:38 ha-866665 kubelet[1369]: E0315 06:35:38.544767 1369 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-provisioner\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=storage-provisioner pod=storage-provisioner_kube-system(b11128b3-f84e-4526-992d-56e278c3f7c9)\"" pod="kube-system/storage-provisioner" podUID="b11128b3-f84e-4526-992d-56e278c3f7c9"
Mar 15 06:35:38 ha-866665 kubelet[1369]: I0315 06:35:38.544900 1369 scope.go:117] "RemoveContainer" containerID="10518fb395ccef0d12367b01f3a67613f203a91a536807a5036c49103719fc54"
Mar 15 06:35:38 ha-866665 kubelet[1369]: E0315 06:35:38.545135 1369 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"kindnet-cni\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=kindnet-cni pod=kindnet-9nvvx_kube-system(4c5333df-bb98-4f27-9197-875a160f4ff6)\"" pod="kube-system/kindnet-9nvvx" podUID="4c5333df-bb98-4f27-9197-875a160f4ff6"
-- /stdout --
** stderr **
E0315 06:35:38.073367 34919 logs.go:258] failed to output last start logs: failed to read file /home/jenkins/minikube-integration/18213-8825/.minikube/logs/lastStart.txt: bufio.Scanner: token too long
** /stderr **
helpers_test.go:254: (dbg) Run: out/minikube-linux-amd64 status --format={{.APIServer}} -p ha-866665 -n ha-866665
helpers_test.go:254: (dbg) Non-zero exit: out/minikube-linux-amd64 status --format={{.APIServer}} -p ha-866665 -n ha-866665: exit status 2 (239.187128ms)
-- stdout --
Stopped
-- /stdout --
helpers_test.go:254: status error: exit status 2 (may be ok)
helpers_test.go:256: "ha-866665" apiserver is not running, skipping kubectl commands (state="Stopped")
--- FAIL: TestMultiControlPlane/serial/DegradedAfterClusterRestart (2.29s)