=== RUN TestCertOptions
=== PAUSE TestCertOptions
=== CONT TestCertOptions
cert_options_test.go:48: (dbg) Run: out/minikube-windows-amd64.exe start -p cert-options-20210915033619-22140 --memory=2048 --apiserver-ips=127.0.0.1 --apiserver-ips=192.168.15.15 --apiserver-names=localhost --apiserver-names=www.google.com --apiserver-port=8555 --driver=docker --apiserver-name=localhost
=== CONT TestCertOptions
cert_options_test.go:48: (dbg) Done: out/minikube-windows-amd64.exe start -p cert-options-20210915033619-22140 --memory=2048 --apiserver-ips=127.0.0.1 --apiserver-ips=192.168.15.15 --apiserver-names=localhost --apiserver-names=www.google.com --apiserver-port=8555 --driver=docker --apiserver-name=localhost: (7m43.7340641s)
cert_options_test.go:59: (dbg) Run: out/minikube-windows-amd64.exe -p cert-options-20210915033619-22140 ssh "openssl x509 -text -noout -in /var/lib/minikube/certs/apiserver.crt"
cert_options_test.go:59: (dbg) Done: out/minikube-windows-amd64.exe -p cert-options-20210915033619-22140 ssh "openssl x509 -text -noout -in /var/lib/minikube/certs/apiserver.crt": (6.1057839s)
cert_options_test.go:74: (dbg) Run: kubectl --context cert-options-20210915033619-22140 config view
cert_options_test.go:79: apiserver server port incorrect. Output of 'kubectl config view' = "\n-- stdout --\n\tapiVersion: v1\n\tclusters:\n\t- cluster:\n\t certificate-authority: C:\\Users\\jenkins\\minikube-integration\\.minikube\\ca.crt\n\t extensions:\n\t - extension:\n\t last-update: Wed, 15 Sep 2021 03:43:28 GMT\n\t provider: minikube.sigs.k8s.io\n\t version: v1.23.0\n\t name: cluster_info\n\t server: https://localhost:58833\n\t name: cert-options-20210915033619-22140\n\t- cluster:\n\t certificate-authority: C:\\Users\\jenkins\\minikube-integration\\.minikube\\ca.crt\n\t extensions:\n\t - extension:\n\t last-update: Wed, 15 Sep 2021 03:43:20 GMT\n\t provider: minikube.sigs.k8s.io\n\t version: v1.23.0\n\t name: cluster_info\n\t server: https://127.0.0.1:58883\n\t name: kubernetes-upgrade-20210915032703-22140\n\t- cluster:\n\t certificate-authority: C:\\Users\\jenkins\\minikube-integration\\.minikube\\ca.crt\n\t server
: https://127.0.0.1:58908\n\t name: missing-upgrade-20210915032655-22140\n\t- cluster:\n\t certificate-authority: C:\\Users\\jenkins\\minikube-integration\\.minikube\\ca.crt\n\t extensions:\n\t - extension:\n\t last-update: Wed, 15 Sep 2021 03:43:40 GMT\n\t provider: minikube.sigs.k8s.io\n\t version: v1.23.0\n\t name: cluster_info\n\t server: https://127.0.0.1:58839\n\t name: old-k8s-version-20210915033621-22140\n\t- cluster:\n\t certificate-authority: C:\\Users\\jenkins\\minikube-integration\\.minikube\\ca.crt\n\t extensions:\n\t - extension:\n\t last-update: Wed, 15 Sep 2021 03:21:11 GMT\n\t provider: minikube.sigs.k8s.io\n\t version: v1.23.0\n\t name: cluster_info\n\t server: https://127.0.0.1:58454\n\t name: pause-20210915030944-22140\n\tcontexts:\n\t- context:\n\t cluster: cert-options-20210915033619-22140\n\t extensions:\n\t - extension:\n\t last-update: Wed, 15 Sep 2021 03:43:28 GMT\n\t provider: minik
ube.sigs.k8s.io\n\t version: v1.23.0\n\t name: context_info\n\t namespace: default\n\t user: cert-options-20210915033619-22140\n\t name: cert-options-20210915033619-22140\n\t- context:\n\t cluster: kubernetes-upgrade-20210915032703-22140\n\t extensions:\n\t - extension:\n\t last-update: Wed, 15 Sep 2021 03:43:20 GMT\n\t provider: minikube.sigs.k8s.io\n\t version: v1.23.0\n\t name: context_info\n\t namespace: default\n\t user: kubernetes-upgrade-20210915032703-22140\n\t name: kubernetes-upgrade-20210915032703-22140\n\t- context:\n\t cluster: missing-upgrade-20210915032655-22140\n\t user: missing-upgrade-20210915032655-22140\n\t name: missing-upgrade-20210915032655-22140\n\t- context:\n\t cluster: old-k8s-version-20210915033621-22140\n\t extensions:\n\t - extension:\n\t last-update: Wed, 15 Sep 2021 03:43:40 GMT\n\t provider: minikube.sigs.k8s.io\n\t version: v1.23.0\n\t name: context_info\n\t namespace: de
fault\n\t user: old-k8s-version-20210915033621-22140\n\t name: old-k8s-version-20210915033621-22140\n\t- context:\n\t cluster: pause-20210915030944-22140\n\t extensions:\n\t - extension:\n\t last-update: Wed, 15 Sep 2021 03:21:11 GMT\n\t provider: minikube.sigs.k8s.io\n\t version: v1.23.0\n\t name: context_info\n\t namespace: default\n\t user: pause-20210915030944-22140\n\t name: pause-20210915030944-22140\n\tcurrent-context: old-k8s-version-20210915033621-22140\n\tkind: Config\n\tpreferences: {}\n\tusers:\n\t- name: cert-options-20210915033619-22140\n\t user:\n\t client-certificate: C:\\Users\\jenkins\\minikube-integration\\.minikube\\profiles\\cert-options-20210915033619-22140\\client.crt\n\t client-key: C:\\Users\\jenkins\\minikube-integration\\.minikube\\profiles\\cert-options-20210915033619-22140\\client.key\n\t- name: kubernetes-upgrade-20210915032703-22140\n\t user:\n\t client-certificate: C:\\Users\\jenkins\\minikube-integration\\.minikube\\prof
iles\\kubernetes-upgrade-20210915032703-22140\\client.crt\n\t client-key: C:\\Users\\jenkins\\minikube-integration\\.minikube\\profiles\\kubernetes-upgrade-20210915032703-22140\\client.key\n\t- name: missing-upgrade-20210915032655-22140\n\t user:\n\t client-certificate: C:\\Users\\jenkins\\minikube-integration\\.minikube\\profiles\\missing-upgrade-20210915032655-22140\\client.crt\n\t client-key: C:\\Users\\jenkins\\minikube-integration\\.minikube\\profiles\\missing-upgrade-20210915032655-22140\\client.key\n\t- name: old-k8s-version-20210915033621-22140\n\t user:\n\t client-certificate: C:\\Users\\jenkins\\minikube-integration\\.minikube\\profiles\\old-k8s-version-20210915033621-22140\\client.crt\n\t client-key: C:\\Users\\jenkins\\minikube-integration\\.minikube\\profiles\\old-k8s-version-20210915033621-22140\\client.key\n\t- name: pause-20210915030944-22140\n\t user:\n\t client-certificate: C:\\Users\\jenkins\\minikube-integration\\.minikube\\profiles\\pause-20210915030944-22140\\client.
crt\n\t client-key: C:\\Users\\jenkins\\minikube-integration\\.minikube\\profiles\\pause-20210915030944-22140\\client.key\n\n-- /stdout --"
cert_options_test.go:82: *** TestCertOptions FAILED at 2021-09-15 03:44:09.7729557 +0000 GMT m=+8151.598442401
helpers_test.go:223: -----------------------post-mortem--------------------------------
helpers_test.go:231: ======> post-mortem[TestCertOptions]: docker inspect <======
helpers_test.go:232: (dbg) Run: docker inspect cert-options-20210915033619-22140
helpers_test.go:236: (dbg) docker inspect cert-options-20210915033619-22140:
-- stdout --
[
{
"Id": "16cb0023c6c8f7aafac0e6d2ae0d44c2d47850c73295e34cfcae20dad980a89f",
"Created": "2021-09-15T03:36:39.4192249Z",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 174420,
"ExitCode": 0,
"Error": "",
"StartedAt": "2021-09-15T03:36:43.4220138Z",
"FinishedAt": "0001-01-01T00:00:00Z"
},
"Image": "sha256:83b5a81388468b1ffcd3874b4f24c1406c63c33ac07797cc8bed6ad0207d36a8",
"ResolvConfPath": "/var/lib/docker/containers/16cb0023c6c8f7aafac0e6d2ae0d44c2d47850c73295e34cfcae20dad980a89f/resolv.conf",
"HostnamePath": "/var/lib/docker/containers/16cb0023c6c8f7aafac0e6d2ae0d44c2d47850c73295e34cfcae20dad980a89f/hostname",
"HostsPath": "/var/lib/docker/containers/16cb0023c6c8f7aafac0e6d2ae0d44c2d47850c73295e34cfcae20dad980a89f/hosts",
"LogPath": "/var/lib/docker/containers/16cb0023c6c8f7aafac0e6d2ae0d44c2d47850c73295e34cfcae20dad980a89f/16cb0023c6c8f7aafac0e6d2ae0d44c2d47850c73295e34cfcae20dad980a89f-json.log",
"Name": "/cert-options-20210915033619-22140",
"RestartCount": 0,
"Driver": "overlay2",
"Platform": "linux",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"ExecIDs": null,
"HostConfig": {
"Binds": [
"/lib/modules:/lib/modules:ro",
"cert-options-20210915033619-22140:/var"
],
"ContainerIDFile": "",
"LogConfig": {
"Type": "json-file",
"Config": {}
},
"NetworkMode": "cert-options-20210915033619-22140",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
],
"8555/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "0"
}
]
},
"RestartPolicy": {
"Name": "no",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": null,
"CapDrop": null,
"CgroupnsMode": "host",
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": null,
"GroupAdd": null,
"IpcMode": "private",
"Cgroup": "",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined",
"apparmor=unconfined",
"label=disable"
],
"Tmpfs": {
"/run": "",
"/tmp": ""
},
"UTSMode": "",
"UsernsMode": "",
"ShmSize": 67108864,
"Runtime": "runc",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 2147483648,
"NanoCpus": 2000000000,
"CgroupParent": "",
"BlkioWeight": 0,
"BlkioWeightDevice": [],
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [
{
"PathOnHost": "/dev/fuse",
"PathInContainer": "/dev/fuse",
"CgroupPermissions": "rwm"
}
],
"DeviceCgroupRules": null,
"DeviceRequests": null,
"KernelMemory": 0,
"KernelMemoryTCP": 0,
"MemoryReservation": 0,
"MemorySwap": 2147483648,
"MemorySwappiness": null,
"OomKillDisable": false,
"PidsLimit": null,
"Ulimits": null,
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"MaskedPaths": null,
"ReadonlyPaths": null
},
"GraphDriver": {
"Data": {
"LowerDir": "/var/lib/docker/overlay2/1d92d4649d4a433116cd85986c2a0a90f64abd137bb3758c87f81abd4a45f810-init/diff:/var/lib/docker/overlay2/81b5ed92bfb1e2a2a0e307c706b587bea810390dd4cdeffdaab53cb2bea532a6/diff:/var/lib/docker/overlay2/9560b70ae747eb38506ca99f7bdf1b19d69a399aa855bf6d066d5631b126dae0/diff:/var/lib/docker/overlay2/695fbfd66132a632f9cf21a1dbf1c4585ecf3d79d4ec664dc7322dbe57733e22/diff:/var/lib/docker/overlay2/db1f669858e6abde6d71803adf0e4dab516d446780d5e6b1fa82ed6e2c992d39/diff:/var/lib/docker/overlay2/fab89974c291c465525b131b7fd3c3d267c0435e58b67e536b1f5e99b0fe3552/diff:/var/lib/docker/overlay2/7d5946148c5ebf869abcd61af8cbd81254b96679a59bff1399fa76d06f970a03/diff:/var/lib/docker/overlay2/ac34ffb8ff292d487d8e0007c602732cac31fc43cc9dd73014f4f7f6731002e4/diff:/var/lib/docker/overlay2/c79772dfc8b60a34db55f8f7bdd7eb21bdb2ae1ebae9e19320eb82d243476de1/diff:/var/lib/docker/overlay2/5f0227571cb11adf4a20233b21288f6215d7ee4baa55da18a29c55f255c3f91b/diff:/var/lib/docker/overlay2/8f8a0a
55c9a3d7643b70fafbe1d581deef7a9142bb7504cade2efea33d17c8b6/diff:/var/lib/docker/overlay2/855d9e351347b1bfa0c8fcdd68ca509489970443ce6ac3f078a84319bbdbb0de/diff:/var/lib/docker/overlay2/d6da6485052539019c636fe8ca30537f92704bc855db6bb09a9228e17d5e5ee1/diff:/var/lib/docker/overlay2/3a712bb22c438ea19740b4d19771cd31cbd08e2f23647daf15e09967798d671d/diff:/var/lib/docker/overlay2/e8f4cc7b40bc0b3a9e62ea0d4f5ca169aab3e908980e13c881a98909769e05a7/diff:/var/lib/docker/overlay2/7364b0516116b13f8d51a574ea9312cc8be87bf0923e8ebe0018085133e57195/diff:/var/lib/docker/overlay2/10d8c9ca18bc3463470c25ce09aa92dc1df0366115c9fd5a22e67d1369e27b72/diff:/var/lib/docker/overlay2/e8ad5dbce212f833465ffdc136c8c744beb3bfe489d7f20f82084f854ab617cd/diff:/var/lib/docker/overlay2/391d7b820cdbb31a7bcc9bd350aff08e83bc2f5083fa09d2d7c1db69d1861b08/diff:/var/lib/docker/overlay2/394198ca9ba772f189cefae2c09414df3798734482a0159958ad4c74374079e8/diff:/var/lib/docker/overlay2/c3620c3c820e1cc79a02390c9ede0beacdc7fe42aa0e9564d27d6c793741eafe/diff:/var/lib/d
ocker/overlay2/9b11f1c010dca16f2c216392f2d3c5ec585e7d2ca91eb0a4824410accaba4ef3/diff:/var/lib/docker/overlay2/d8e94cabdfcf34c1c2ecb5355519daea41ba85e90131944f14c6c5faadb3f538/diff:/var/lib/docker/overlay2/335c17cc3e6bcc49659f681fefa84f63f496fab770f62dd31577690f8e3958b6/diff:/var/lib/docker/overlay2/5ef44871aef3ad96e532fdbc78e5379afd65c7ffd39bed734ed35daf134257b5/diff:/var/lib/docker/overlay2/ce73bde16589364238c0bb925bbd93f9b2b9c5e2f3267cc196298f62fbc08342/diff:/var/lib/docker/overlay2/461113b8bc693d226593885e543b82eac9a75ea77d0bcdaa60551cca12495538/diff:/var/lib/docker/overlay2/f7d47793cf5882d3e0b92ebb0d7d2456fc621d6db83cb2439f96c4b248b11d25/diff:/var/lib/docker/overlay2/a8e74e4377f38c1a50d9a335bfc92405a4df112abdcbd2555cbe3b592f071fd5/diff:/var/lib/docker/overlay2/405812e0a303b666cd7c1c0102d8f415494b9641e1f5ab9404e146c2265592cb/diff:/var/lib/docker/overlay2/deecfc978d174b5d2c0a209b450d0fa15828234099690cc9092c6ff67a1926d2/diff:/var/lib/docker/overlay2/6fa41c9e75c99fb82729fdd55e5653ce5b7edf256a1dd8791c3012cf210
7f486/diff:/var/lib/docker/overlay2/2dd2dde99da44abd645912f40fdb7d06e201a622cccf049222fa9a53ab6ca234/diff:/var/lib/docker/overlay2/a73187a91c6737ec4627be55f4b58dab9d4ef30412857cbf1cd6e6778962c9f4/diff:/var/lib/docker/overlay2/7fcd2796c0a1717ddf6c90aad88aff2e11a87b836d8761e756b6bc7a292ed570/diff:/var/lib/docker/overlay2/276597df229fc32d0d371563f135664fa4bef3fbc20372998b7b051504e6188a/diff:/var/lib/docker/overlay2/28f6cf4ea77b5f1df2373079b5b3c9b2ec7e95488cec51c54e7ff22f8fea2f36/diff:/var/lib/docker/overlay2/301627855ef95ac8b04f9b404290e80b6a94b9637ec2ca0c31b5701c6ac786fd/diff:/var/lib/docker/overlay2/a589a72c723642d2bb727fead8edfcaffaca10eed1bb4af32fac19fb6fc32874/diff:/var/lib/docker/overlay2/90d1c9e6fe8a1c74ac53d78f9a0b7ee36fc624becac59c2a6056c004ebe45e1e/diff",
"MergedDir": "/var/lib/docker/overlay2/1d92d4649d4a433116cd85986c2a0a90f64abd137bb3758c87f81abd4a45f810/merged",
"UpperDir": "/var/lib/docker/overlay2/1d92d4649d4a433116cd85986c2a0a90f64abd137bb3758c87f81abd4a45f810/diff",
"WorkDir": "/var/lib/docker/overlay2/1d92d4649d4a433116cd85986c2a0a90f64abd137bb3758c87f81abd4a45f810/work"
},
"Name": "overlay2"
},
"Mounts": [
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Mode": "ro",
"RW": false,
"Propagation": "rprivate"
},
{
"Type": "volume",
"Name": "cert-options-20210915033619-22140",
"Source": "/var/lib/docker/volumes/cert-options-20210915033619-22140/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "z",
"RW": true,
"Propagation": ""
}
],
"Config": {
"Hostname": "cert-options-20210915033619-22140",
"Domainname": "",
"User": "root",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"ExposedPorts": {
"22/tcp": {},
"2376/tcp": {},
"32443/tcp": {},
"5000/tcp": {},
"8555/tcp": {}
},
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=docker",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase-builds:v0.0.26-1631295795-12425@sha256:7d61c0b6cf6832c8015ada78640635c5ab74b72f12f51bcc4c7660b0be01af56",
"Volumes": null,
"WorkingDir": "",
"Entrypoint": [
"/usr/local/bin/entrypoint",
"/sbin/init"
],
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "cert-options-20210915033619-22140",
"name.minikube.sigs.k8s.io": "cert-options-20210915033619-22140",
"role.minikube.sigs.k8s.io": ""
},
"StopSignal": "SIGRTMIN+3"
},
"NetworkSettings": {
"Bridge": "",
"SandboxID": "faddf7fbefe21a28a0248b2ee72634bc1121120a17441089fffa252b1e5027cf",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "58834"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "58830"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "58831"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "58832"
}
],
"8555/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "58833"
}
]
},
"SandboxKey": "/var/run/docker/netns/faddf7fbefe2",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"cert-options-20210915033619-22140": {
"IPAMConfig": {
"IPv4Address": "192.168.58.2"
},
"Links": null,
"Aliases": [
"16cb0023c6c8",
"cert-options-20210915033619-22140"
],
"NetworkID": "3813cf20a9594cef7b7cdde6d44e1b11f2701ba643a655276e6377e8398861d5",
"EndpointID": "d005455b98f5d2c105220c19280648e69f992223f3d3643d733b92a2ad5c2470",
"Gateway": "192.168.58.1",
"IPAddress": "192.168.58.2",
"IPPrefixLen": 24,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:c0:a8:3a:02",
"DriverOpts": null
}
}
}
}
]
-- /stdout --
helpers_test.go:240: (dbg) Run: out/minikube-windows-amd64.exe status --format={{.Host}} -p cert-options-20210915033619-22140 -n cert-options-20210915033619-22140
helpers_test.go:240: (dbg) Done: out/minikube-windows-amd64.exe status --format={{.Host}} -p cert-options-20210915033619-22140 -n cert-options-20210915033619-22140: (8.5369469s)
helpers_test.go:245: <<< TestCertOptions FAILED: start of post-mortem logs <<<
helpers_test.go:246: ======> post-mortem[TestCertOptions]: minikube logs <======
helpers_test.go:248: (dbg) Run: out/minikube-windows-amd64.exe -p cert-options-20210915033619-22140 logs -n 25
helpers_test.go:248: (dbg) Done: out/minikube-windows-amd64.exe -p cert-options-20210915033619-22140 logs -n 25: (41.1543122s)
helpers_test.go:253: TestCertOptions logs:
-- stdout --
*
* ==> Audit <==
* |---------|-----------------------------------------|-----------------------------------------|-------------------------|---------|-------------------------------|-------------------------------|
| Command | Args | Profile | User | Version | Start Time | End Time |
|---------|-----------------------------------------|-----------------------------------------|-------------------------|---------|-------------------------------|-------------------------------|
| start | -p pause-20210915030944-22140 | pause-20210915030944-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:20:10 GMT | Wed, 15 Sep 2021 03:21:42 GMT |
| | --alsologtostderr -v=1 | | | | | |
| | --driver=docker | | | | | |
| pause | -p pause-20210915030944-22140 | pause-20210915030944-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:21:43 GMT | Wed, 15 Sep 2021 03:22:01 GMT |
| | --alsologtostderr -v=5 | | | | | |
| unpause | -p pause-20210915030944-22140 | pause-20210915030944-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:22:46 GMT | Wed, 15 Sep 2021 03:22:59 GMT |
| | --alsologtostderr -v=5 | | | | | |
| pause | -p pause-20210915030944-22140 | pause-20210915030944-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:22:59 GMT | Wed, 15 Sep 2021 03:23:23 GMT |
| | --alsologtostderr -v=5 | | | | | |
| start | -p | stopped-upgrade-20210915030944-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:22:19 GMT | Wed, 15 Sep 2021 03:26:03 GMT |
| | stopped-upgrade-20210915030944-22140 | | | | | |
| | --memory=2200 --alsologtostderr -v=1 | | | | | |
| | --driver=docker | | | | | |
| start | -p | running-upgrade-20210915030944-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:22:40 GMT | Wed, 15 Sep 2021 03:26:17 GMT |
| | running-upgrade-20210915030944-22140 | | | | | |
| | --memory=2200 --alsologtostderr -v=1 | | | | | |
| | --driver=docker | | | | | |
| logs | -p | stopped-upgrade-20210915030944-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:26:03 GMT | Wed, 15 Sep 2021 03:26:23 GMT |
| | stopped-upgrade-20210915030944-22140 | | | | | |
| start | -p | force-systemd-flag-20210915032047-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:20:47 GMT | Wed, 15 Sep 2021 03:26:47 GMT |
| | force-systemd-flag-20210915032047-22140 | | | | | |
| | --memory=2048 --force-systemd | | | | | |
| | --alsologtostderr -v=5 --driver=docker | | | | | |
| delete | -p | running-upgrade-20210915030944-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:26:17 GMT | Wed, 15 Sep 2021 03:26:50 GMT |
| | running-upgrade-20210915030944-22140 | | | | | |
| -p | force-systemd-flag-20210915032047-22140 | force-systemd-flag-20210915032047-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:26:48 GMT | Wed, 15 Sep 2021 03:26:54 GMT |
| | ssh docker info --format | | | | | |
| | {{.CgroupDriver}} | | | | | |
| delete | -p | stopped-upgrade-20210915030944-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:26:24 GMT | Wed, 15 Sep 2021 03:26:54 GMT |
| | stopped-upgrade-20210915030944-22140 | | | | | |
| delete | -p | flannel-20210915032655-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:26:55 GMT | Wed, 15 Sep 2021 03:27:03 GMT |
| | flannel-20210915032655-22140 | | | | | |
| delete | -p | force-systemd-flag-20210915032047-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:26:54 GMT | Wed, 15 Sep 2021 03:27:27 GMT |
| | force-systemd-flag-20210915032047-22140 | | | | | |
| start | -p | force-systemd-env-20210915032650-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:26:50 GMT | Wed, 15 Sep 2021 03:35:36 GMT |
| | force-systemd-env-20210915032650-22140 | | | | | |
| | --memory=2048 --alsologtostderr -v=5 | | | | | |
| | --driver=docker | | | | | |
| start | -p | docker-flags-20210915032727-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:27:27 GMT | Wed, 15 Sep 2021 03:35:42 GMT |
| | docker-flags-20210915032727-22140 | | | | | |
| | --cache-images=false | | | | | |
| | --memory=2048 | | | | | |
| | --install-addons=false | | | | | |
| | --wait=false --docker-env=FOO=BAR | | | | | |
| | --docker-env=BAZ=BAT | | | | | |
| | --docker-opt=debug | | | | | |
| | --docker-opt=icc=true | | | | | |
| | --alsologtostderr -v=5 | | | | | |
| | --driver=docker | | | | | |
| -p | force-systemd-env-20210915032650-22140 | force-systemd-env-20210915032650-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:35:37 GMT | Wed, 15 Sep 2021 03:35:46 GMT |
| | ssh docker info --format | | | | | |
| | {{.CgroupDriver}} | | | | | |
| -p | docker-flags-20210915032727-22140 | docker-flags-20210915032727-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:35:42 GMT | Wed, 15 Sep 2021 03:35:49 GMT |
| | ssh sudo systemctl show docker | | | | | |
| | --property=Environment --no-pager | | | | | |
| -p | docker-flags-20210915032727-22140 | docker-flags-20210915032727-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:35:50 GMT | Wed, 15 Sep 2021 03:35:54 GMT |
| | ssh sudo systemctl show docker | | | | | |
| | --property=ExecStart --no-pager | | | | | |
| delete | -p | force-systemd-env-20210915032650-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:35:47 GMT | Wed, 15 Sep 2021 03:36:19 GMT |
| | force-systemd-env-20210915032650-22140 | | | | | |
| delete | -p | docker-flags-20210915032727-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:35:55 GMT | Wed, 15 Sep 2021 03:36:21 GMT |
| | docker-flags-20210915032727-22140 | | | | | |
| start | -p | kubernetes-upgrade-20210915032703-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:27:04 GMT | Wed, 15 Sep 2021 03:37:04 GMT |
| | kubernetes-upgrade-20210915032703-22140 | | | | | |
| | --memory=2200 | | | | | |
| | --kubernetes-version=v1.14.0 | | | | | |
| | --alsologtostderr -v=1 --driver=docker | | | | | |
| stop | -p | kubernetes-upgrade-20210915032703-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:37:04 GMT | Wed, 15 Sep 2021 03:37:18 GMT |
| | kubernetes-upgrade-20210915032703-22140 | | | | | |
| start | -p | kubernetes-upgrade-20210915032703-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:37:20 GMT | Wed, 15 Sep 2021 03:43:53 GMT |
| | kubernetes-upgrade-20210915032703-22140 | | | | | |
| | --memory=2200 | | | | | |
| | --kubernetes-version=v1.22.2-rc.0 | | | | | |
| | --alsologtostderr -v=1 --driver=docker | | | | | |
| start | -p | cert-options-20210915033619-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:36:20 GMT | Wed, 15 Sep 2021 03:44:03 GMT |
| | cert-options-20210915033619-22140 | | | | | |
| | --memory=2048 | | | | | |
| | --apiserver-ips=127.0.0.1 | | | | | |
| | --apiserver-ips=192.168.15.15 | | | | | |
| | --apiserver-names=localhost | | | | | |
| | --apiserver-names=www.google.com | | | | | |
| | --apiserver-port=8555 | | | | | |
| | --driver=docker | | | | | |
| | --apiserver-name=localhost | | | | | |
| -p | cert-options-20210915033619-22140 | cert-options-20210915033619-22140 | WINDOWS-SERVER-\jenkins | v1.23.0 | Wed, 15 Sep 2021 03:44:03 GMT | Wed, 15 Sep 2021 03:44:09 GMT |
| | ssh openssl x509 -text -noout -in | | | | | |
| | /var/lib/minikube/certs/apiserver.crt | | | | | |
|---------|-----------------------------------------|-----------------------------------------|-------------------------|---------|-------------------------------|-------------------------------|
*
* ==> Last Start <==
* Log file created at: 2021/09/15 03:43:54
Running on machine: windows-server-1
Binary: Built with gc go1.17 for windows/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0915 03:43:54.969151 1756 out.go:298] Setting OutFile to fd 1236 ...
I0915 03:43:54.970184 1756 out.go:345] TERM=,COLORTERM=, which probably does not support color
I0915 03:43:54.970184 1756 out.go:311] Setting ErrFile to fd 972...
I0915 03:43:54.970184 1756 out.go:345] TERM=,COLORTERM=, which probably does not support color
I0915 03:43:54.999383 1756 out.go:305] Setting JSON to false
I0915 03:43:55.012211 1756 start.go:111] hostinfo: {"hostname":"windows-server-1","uptime":10280217,"bootTime":1621397217,"procs":159,"os":"windows","platform":"Microsoft Windows Server 2019 Datacenter","platformFamily":"Server","platformVersion":"10.0.17763 Build 17763","kernelVersion":"10.0.17763 Build 17763","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"9879231f-6171-435d-bab4-5b366cc6391b"}
W0915 03:43:55.013047 1756 start.go:119] gopshost.Virtualization returned error: not implemented yet
I0915 03:43:55.021358 1756 out.go:177] * [kubernetes-upgrade-20210915032703-22140] minikube v1.23.0 on Microsoft Windows Server 2019 Datacenter 10.0.17763 Build 17763
I0915 03:43:55.021689 1756 notify.go:169] Checking for updates...
I0915 03:43:55.033155 1756 out.go:177] - KUBECONFIG=C:\Users\jenkins\minikube-integration\kubeconfig
I0915 03:43:55.035750 1756 out.go:177] - MINIKUBE_HOME=C:\Users\jenkins\minikube-integration\.minikube
I0915 03:43:52.053482 50104 pod_ready.go:102] pod "coredns-fb8b8dccf-kkhkd" in "kube-system" namespace has status "Ready":"False"
I0915 03:43:54.115114 50104 pod_ready.go:102] pod "coredns-fb8b8dccf-kkhkd" in "kube-system" namespace has status "Ready":"False"
I0915 03:43:56.542703 50104 pod_ready.go:102] pod "coredns-fb8b8dccf-kkhkd" in "kube-system" namespace has status "Ready":"False"
I0915 03:43:55.037883 1756 out.go:177] - MINIKUBE_LOCATION=12425
I0915 03:43:55.039125 1756 config.go:177] Loaded profile config "kubernetes-upgrade-20210915032703-22140": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.22.2-rc.0
I0915 03:43:55.040366 1756 driver.go:343] Setting default libvirt URI to qemu:///system
I0915 03:43:55.763900 1756 docker.go:132] docker version: linux-20.10.5
I0915 03:43:55.784660 1756 cli_runner.go:115] Run: docker system info --format "{{json .}}"
I0915 03:43:57.120738 1756 cli_runner.go:168] Completed: docker system info --format "{{json .}}": (1.3351921s)
I0915 03:43:57.121592 1756 info.go:263] docker info: {ID:6FWJ:GOIP:3UEJ:4EPV:BN5V:RES7:2PF6:QH5I:B3LP:YJP2:JQEM:LHWQ Containers:5 ContainersRunning:5 ContainersPaused:0 ContainersStopped:0 Images:7 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:84 OomKillDisable:true NGoroutines:79 SystemTime:2021-09-15 03:43:56.5156682 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:4 KernelVersion:4.19.121-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServerAddress:https://inde
x.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:4 MemTotal:20973547520 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:20.10.5 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:269548fa27e0089a8b8278fc4fc781d7f65a939b Expected:269548fa27e0089a8b8278fc4fc781d7f65a939b} RuncCommit:{ID:ff819c7e9184c13b7c2607fe6c30ae19403a7aff Expected:ff819c7e9184c13b7c2607fe6c30ae19403a7aff} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=default] ProductLicense: Warnings:<nil> ServerErrors:[]
ClientInfo:{Debug:false Plugins:[map[Experimental:true Name:app Path:C:\ProgramData\Docker\cli-plugins\docker-app.exe SchemaVersion:0.1.0 ShortDescription:Docker App Vendor:Docker Inc. Version:v0.9.1-beta3] map[Name:buildx Path:C:\ProgramData\Docker\cli-plugins\docker-buildx.exe SchemaVersion:0.1.0 ShortDescription:Build with BuildKit Vendor:Docker Inc. Version:v0.5.1-docker] map[Name:scan Path:C:\ProgramData\Docker\cli-plugins\docker-scan.exe SchemaVersion:0.1.0 ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.5.0]] Warnings:<nil>}}
I0915 03:43:53.398453 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:43:53.911054 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:43:54.397390 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:43:54.898883 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:43:55.403816 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:43:55.904422 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:43:56.400300 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:43:56.902021 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:43:57.400219 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:43:57.899254 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:43:57.130668 1756 out.go:177] * Using the docker driver based on existing profile
I0915 03:43:57.130996 1756 start.go:278] selected driver: docker
I0915 03:43:57.131218 1756 start.go:751] validating driver "docker" against &{Name:kubernetes-upgrade-20210915032703-22140 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.26-1631295795-12425@sha256:7d61c0b6cf6832c8015ada78640635c5ab74b72f12f51bcc4c7660b0be01af56 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.22.2-rc.0 ClusterName:kubernetes-upgrade-20210915032703-22140 Names
pace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.22.2-rc.0 ControlPlane:true Worker:true}] Addons:map[ambassador:false auto-pause:false csi-hostpath-driver:false dashboard:false default-storageclass:true efk:false freshpod:false gcp-auth:false gvisor:false helm-tiller:false ingress:false ingress-dns:false istio:false istio-provisioner:false kubevirt:false logviewer:false metallb:false metrics-server:false nvidia-driver-installer:false nvidia-gpu-device-plugin:false olm:false pod-security-policy:false portainer:false registry:false registry-aliases:false registry-creds:false storage-provisioner:true storage-provisioner-gluster:false volume
snapshots:false] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false ExtraDisks:0}
I0915 03:43:57.131441 1756 start.go:762] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc:}
I0915 03:43:57.245402 1756 cli_runner.go:115] Run: docker system info --format "{{json .}}"
I0915 03:43:58.383076 1756 cli_runner.go:168] Completed: docker system info --format "{{json .}}": (1.1370899s)
I0915 03:43:58.383853 1756 info.go:263] docker info: {ID:6FWJ:GOIP:3UEJ:4EPV:BN5V:RES7:2PF6:QH5I:B3LP:YJP2:JQEM:LHWQ Containers:5 ContainersRunning:5 ContainersPaused:0 ContainersStopped:0 Images:7 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true]] SystemStatus:<nil> Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization:<nil> Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:84 OomKillDisable:true NGoroutines:79 SystemTime:2021-09-15 03:43:57.8767429 +0000 UTC LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:4 KernelVersion:4.19.121-linuxkit OperatingSystem:Docker Desktop OSType:linux Architecture:x86_64 IndexServerAddress:https://inde
x.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:4 MemTotal:20973547520 GenericResources:<nil> DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:docker-desktop Labels:[] ExperimentalBuild:false ServerVersion:20.10.5 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:<nil>} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:269548fa27e0089a8b8278fc4fc781d7f65a939b Expected:269548fa27e0089a8b8278fc4fc781d7f65a939b} RuncCommit:{ID:ff819c7e9184c13b7c2607fe6c30ae19403a7aff Expected:ff819c7e9184c13b7c2607fe6c30ae19403a7aff} InitCommit:{ID:de40ad0 Expected:de40ad0} SecurityOptions:[name=seccomp,profile=default] ProductLicense: Warnings:<nil> ServerErrors:[]
ClientInfo:{Debug:false Plugins:[map[Experimental:true Name:app Path:C:\ProgramData\Docker\cli-plugins\docker-app.exe SchemaVersion:0.1.0 ShortDescription:Docker App Vendor:Docker Inc. Version:v0.9.1-beta3] map[Name:buildx Path:C:\ProgramData\Docker\cli-plugins\docker-buildx.exe SchemaVersion:0.1.0 ShortDescription:Build with BuildKit Vendor:Docker Inc. Version:v0.5.1-docker] map[Name:scan Path:C:\ProgramData\Docker\cli-plugins\docker-scan.exe SchemaVersion:0.1.0 ShortDescription:Docker Scan Vendor:Docker Inc. Version:v0.5.0]] Warnings:<nil>}}
I0915 03:43:58.384801 1756 cni.go:93] Creating CNI manager for ""
I0915 03:43:58.384801 1756 cni.go:167] CNI unnecessary in this configuration, recommending no CNI
I0915 03:43:58.384801 1756 start_flags.go:278] config:
{Name:kubernetes-upgrade-20210915032703-22140 KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.26-1631295795-12425@sha256:7d61c0b6cf6832c8015ada78640635c5ab74b72f12f51bcc4c7660b0be01af56 Memory:2200 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.22.2-rc.0 ClusterName:kubernetes-upgrade-20210915032703-22140 Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomai
n:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.76.2 Port:8443 KubernetesVersion:v1.22.2-rc.0 ControlPlane:true Worker:true}] Addons:map[ambassador:false auto-pause:false csi-hostpath-driver:false dashboard:false default-storageclass:true efk:false freshpod:false gcp-auth:false gvisor:false helm-tiller:false ingress:false ingress-dns:false istio:false istio-provisioner:false kubevirt:false logviewer:false metallb:false metrics-server:false nvidia-driver-installer:false nvidia-gpu-device-plugin:false olm:false pod-security-policy:false portainer:false registry:false registry-aliases:false registry-creds:false storage-provisioner:true storage-provisioner-gluster:false volumesnapshots:false] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyCompo
nents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false ExtraDisks:0}
I0915 03:43:58.389389 1756 out.go:177] * Starting control plane node kubernetes-upgrade-20210915032703-22140 in cluster kubernetes-upgrade-20210915032703-22140
I0915 03:43:58.389808 1756 cache.go:118] Beginning downloading kic base image for docker with docker
I0915 03:43:58.392241 1756 out.go:177] * Pulling base image ...
I0915 03:43:58.392715 1756 preload.go:131] Checking if preload exists for k8s version v1.22.2-rc.0 and runtime docker
I0915 03:43:58.392715 1756 image.go:75] Checking for gcr.io/k8s-minikube/kicbase-builds:v0.0.26-1631295795-12425@sha256:7d61c0b6cf6832c8015ada78640635c5ab74b72f12f51bcc4c7660b0be01af56 in local docker daemon
I0915 03:43:58.393241 1756 preload.go:147] Found local preload: C:\Users\jenkins\minikube-integration\.minikube\cache\preloaded-tarball\preloaded-images-k8s-v12-v1.22.2-rc.0-docker-overlay2-amd64.tar.lz4
I0915 03:43:58.393412 1756 cache.go:57] Caching tarball of preloaded images
I0915 03:43:58.394644 1756 preload.go:173] Found C:\Users\jenkins\minikube-integration\.minikube\cache\preloaded-tarball\preloaded-images-k8s-v12-v1.22.2-rc.0-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0915 03:43:58.394924 1756 cache.go:60] Finished verifying existence of preloaded tar for v1.22.2-rc.0 on docker
I0915 03:43:58.395341 1756 profile.go:148] Saving config to C:\Users\jenkins\minikube-integration\.minikube\profiles\kubernetes-upgrade-20210915032703-22140\config.json ...
I0915 03:43:59.171890 1756 image.go:79] Found gcr.io/k8s-minikube/kicbase-builds:v0.0.26-1631295795-12425@sha256:7d61c0b6cf6832c8015ada78640635c5ab74b72f12f51bcc4c7660b0be01af56 in local docker daemon, skipping pull
I0915 03:43:59.171890 1756 cache.go:140] gcr.io/k8s-minikube/kicbase-builds:v0.0.26-1631295795-12425@sha256:7d61c0b6cf6832c8015ada78640635c5ab74b72f12f51bcc4c7660b0be01af56 exists in daemon, skipping load
I0915 03:43:59.172210 1756 cache.go:206] Successfully downloaded all kic artifacts
I0915 03:43:59.172938 1756 start.go:313] acquiring machines lock for kubernetes-upgrade-20210915032703-22140: {Name:mk7af999dfd2d3dc1bf447f052ae1725ebafaa2f Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0915 03:43:59.173452 1756 start.go:317] acquired machines lock for "kubernetes-upgrade-20210915032703-22140" in 261.8µs
I0915 03:43:59.173761 1756 start.go:93] Skipping create...Using existing machine configuration
I0915 03:43:59.173761 1756 fix.go:55] fixHost starting:
I0915 03:43:59.209526 1756 cli_runner.go:115] Run: docker container inspect kubernetes-upgrade-20210915032703-22140 --format={{.State.Status}}
I0915 03:44:00.041690 1756 fix.go:108] recreateIfNeeded on kubernetes-upgrade-20210915032703-22140: state=Running err=<nil>
W0915 03:44:00.042108 1756 fix.go:134] unexpected machine state, will restart: <nil>
I0915 03:43:58.656965 50104 pod_ready.go:102] pod "coredns-fb8b8dccf-kkhkd" in "kube-system" namespace has status "Ready":"False"
I0915 03:44:01.070043 50104 pod_ready.go:102] pod "coredns-fb8b8dccf-kkhkd" in "kube-system" namespace has status "Ready":"False"
I0915 03:44:02.456932 26504 ssh_runner.go:192] Completed: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.22.1/kubectl apply -f /etc/kubernetes/addons/storageclass.yaml: (26.4618059s)
I0915 03:44:02.456932 26504 ssh_runner.go:192] Completed: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.22.1/kubectl --kubeconfig=/var/lib/minikube/kubeconfig -n kube-system get configmap coredns -o yaml | sed '/^ forward . \/etc\/resolv.conf.*/i \ hosts {\n 192.168.65.2 host.minikube.internal\n fallthrough\n }' | sudo /var/lib/minikube/binaries/v1.22.1/kubectl --kubeconfig=/var/lib/minikube/kubeconfig replace -f -": (26.2801787s)
I0915 03:44:02.456932 26504 ssh_runner.go:192] Completed: sudo KUBECONFIG=/var/lib/minikube/kubeconfig /var/lib/minikube/binaries/v1.22.1/kubectl apply -f /etc/kubernetes/addons/storage-provisioner.yaml: (26.9669015s)
I0915 03:44:02.456932 26504 start.go:729] {"host.minikube.internal": 192.168.65.2} host record injected into CoreDNS
I0915 03:44:02.456932 26504 ssh_runner.go:192] Completed: sudo pgrep -xnf kube-apiserver.*minikube.*: (25.4825054s)
I0915 03:44:02.456932 26504 api_server.go:70] duration metric: took 32.5831642s to wait for apiserver process to appear ...
I0915 03:44:02.456932 26504 api_server.go:86] waiting for apiserver healthz status ...
I0915 03:44:02.456932 26504 api_server.go:239] Checking apiserver healthz at https://localhost:58833/healthz ...
I0915 03:43:58.420345 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:43:58.898663 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:43:59.402714 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:43:59.898991 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:00.406499 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:00.901083 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:01.399445 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:01.904689 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:02.399301 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:02.909626 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:02.464675 26504 out.go:177] * Enabled addons: default-storageclass, storage-provisioner
I0915 03:44:02.464815 26504 addons.go:406] enableAddons completed in 32.5910465s
I0915 03:44:02.848990 26504 api_server.go:265] https://localhost:58833/healthz returned 200:
ok
I0915 03:44:02.865623 26504 api_server.go:139] control plane version: v1.22.1
I0915 03:44:02.865623 26504 api_server.go:129] duration metric: took 408.6924ms to wait for apiserver health ...
I0915 03:44:02.865878 26504 system_pods.go:43] waiting for kube-system pods to appear ...
I0915 03:44:02.962446 26504 system_pods.go:59] 7 kube-system pods found
I0915 03:44:02.962446 26504 system_pods.go:61] "coredns-78fcd69978-cb8sj" [6c354c34-6a2e-45c2-ab92-5442b51a8b82] Pending
I0915 03:44:02.962446 26504 system_pods.go:61] "etcd-cert-options-20210915033619-22140" [89fbf87c-fa61-4ec0-925f-fc79faed7c0f] Pending
I0915 03:44:02.962446 26504 system_pods.go:61] "kube-apiserver-cert-options-20210915033619-22140" [1626db0a-3007-4623-b52d-3d23496cfc78] Pending
I0915 03:44:02.962446 26504 system_pods.go:61] "kube-controller-manager-cert-options-20210915033619-22140" [1e7ecd28-87dc-4044-9b22-dc9ec8a0030f] Running / Ready:ContainersNotReady (containers with unready status: [kube-controller-manager]) / ContainersReady:ContainersNotReady (containers with unready status: [kube-controller-manager])
I0915 03:44:02.962446 26504 system_pods.go:61] "kube-proxy-wmzsf" [cea7699e-ab59-4817-88b4-afda2970424d] Pending / Ready:ContainersNotReady (containers with unready status: [kube-proxy]) / ContainersReady:ContainersNotReady (containers with unready status: [kube-proxy])
I0915 03:44:02.962628 26504 system_pods.go:61] "kube-scheduler-cert-options-20210915033619-22140" [278a8107-bab1-4c53-8aec-710e46a04b4c] Running / Ready:ContainersNotReady (containers with unready status: [kube-scheduler]) / ContainersReady:ContainersNotReady (containers with unready status: [kube-scheduler])
I0915 03:44:02.962628 26504 system_pods.go:61] "storage-provisioner" [42ce7029-cc6b-4bb9-b68b-befa68dc6449] Pending
I0915 03:44:02.962628 26504 system_pods.go:74] duration metric: took 96.75ms to wait for pod list to return data ...
I0915 03:44:02.962628 26504 kubeadm.go:547] duration metric: took 33.0888611s to wait for : map[apiserver:true system_pods:true] ...
I0915 03:44:02.962628 26504 node_conditions.go:102] verifying NodePressure condition ...
I0915 03:44:03.106433 26504 node_conditions.go:122] node storage ephemeral capacity is 65792556Ki
I0915 03:44:03.106511 26504 node_conditions.go:123] node cpu capacity is 4
I0915 03:44:03.106511 26504 node_conditions.go:105] duration metric: took 143.8843ms to run NodePressure ...
I0915 03:44:03.106511 26504 start.go:231] waiting for startup goroutines ...
I0915 03:44:03.306110 26504 start.go:462] kubectl: 1.20.0, cluster: 1.22.1 (minor skew: 2)
I0915 03:44:03.308783 26504 out.go:177]
W0915 03:44:03.308783 26504 out.go:242] ! C:\Program Files\Docker\Docker\resources\bin\kubectl.exe is version 1.20.0, which may have incompatibilites with Kubernetes 1.22.1.
I0915 03:44:03.311917 26504 out.go:177] - Want kubectl v1.22.1? Try 'minikube kubectl -- get pods -A'
I0915 03:44:03.314869 26504 out.go:177] * Done! kubectl is now configured to use "cert-options-20210915033619-22140" cluster and "default" namespace by default
I0915 03:44:00.045099 1756 out.go:177] * Updating the running docker "kubernetes-upgrade-20210915032703-22140" container ...
I0915 03:44:00.045316 1756 machine.go:88] provisioning docker machine ...
I0915 03:44:00.045500 1756 ubuntu.go:169] provisioning hostname "kubernetes-upgrade-20210915032703-22140"
I0915 03:44:00.054068 1756 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" kubernetes-upgrade-20210915032703-22140
I0915 03:44:00.867839 1756 main.go:130] libmachine: Using SSH client type: native
I0915 03:44:00.867839 1756 main.go:130] libmachine: &{{{<nil> 0 [] [] []} docker [0x158acc0] 0x158db80 <nil> [] 0s} 127.0.0.1 58879 <nil> <nil>}
I0915 03:44:00.867839 1756 main.go:130] libmachine: About to run SSH command:
sudo hostname kubernetes-upgrade-20210915032703-22140 && echo "kubernetes-upgrade-20210915032703-22140" | sudo tee /etc/hostname
I0915 03:44:03.862282 1756 main.go:130] libmachine: SSH cmd err, output: <nil>: kubernetes-upgrade-20210915032703-22140
I0915 03:44:03.874249 1756 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" kubernetes-upgrade-20210915032703-22140
I0915 03:44:03.599527 50104 pod_ready.go:102] pod "coredns-fb8b8dccf-kkhkd" in "kube-system" namespace has status "Ready":"False"
I0915 03:44:05.815901 50104 pod_ready.go:102] pod "coredns-fb8b8dccf-kkhkd" in "kube-system" namespace has status "Ready":"False"
I0915 03:44:03.400451 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:03.899950 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:04.905930 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:05.412458 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:05.904122 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:06.417752 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:06.905514 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:07.400071 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:07.897716 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:04.749155 1756 main.go:130] libmachine: Using SSH client type: native
I0915 03:44:04.749969 1756 main.go:130] libmachine: &{{{<nil> 0 [] [] []} docker [0x158acc0] 0x158db80 <nil> [] 0s} 127.0.0.1 58879 <nil> <nil>}
I0915 03:44:04.749969 1756 main.go:130] libmachine: About to run SSH command:
if ! grep -xq '.*\skubernetes-upgrade-20210915032703-22140' /etc/hosts; then
if grep -xq '127.0.1.1\s.*' /etc/hosts; then
sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 kubernetes-upgrade-20210915032703-22140/g' /etc/hosts;
else
echo '127.0.1.1 kubernetes-upgrade-20210915032703-22140' | sudo tee -a /etc/hosts;
fi
fi
I0915 03:44:07.590989 1756 main.go:130] libmachine: SSH cmd err, output: <nil>:
I0915 03:44:07.591162 1756 ubuntu.go:175] set auth options {CertDir:C:\Users\jenkins\minikube-integration\.minikube CaCertPath:C:\Users\jenkins\minikube-integration\.minikube\certs\ca.pem CaPrivateKeyPath:C:\Users\jenkins\minikube-integration\.minikube\certs\ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:C:\Users\jenkins\minikube-integration\.minikube\machines\server.pem ServerKeyPath:C:\Users\jenkins\minikube-integration\.minikube\machines\server-key.pem ClientKeyPath:C:\Users\jenkins\minikube-integration\.minikube\certs\key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:C:\Users\jenkins\minikube-integration\.minikube\certs\cert.pem ServerCertSANs:[] StorePath:C:\Users\jenkins\minikube-integration\.minikube}
I0915 03:44:07.591162 1756 ubuntu.go:177] setting up certificates
I0915 03:44:07.591162 1756 provision.go:83] configureAuth start
I0915 03:44:07.602203 1756 cli_runner.go:115] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" kubernetes-upgrade-20210915032703-22140
I0915 03:44:08.416054 1756 provision.go:138] copyHostCerts
I0915 03:44:08.416834 1756 exec_runner.go:145] found C:\Users\jenkins\minikube-integration\.minikube/key.pem, removing ...
I0915 03:44:08.416834 1756 exec_runner.go:208] rm: C:\Users\jenkins\minikube-integration\.minikube\key.pem
I0915 03:44:08.417446 1756 exec_runner.go:152] cp: C:\Users\jenkins\minikube-integration\.minikube\certs\key.pem --> C:\Users\jenkins\minikube-integration\.minikube/key.pem (1679 bytes)
I0915 03:44:08.420621 1756 exec_runner.go:145] found C:\Users\jenkins\minikube-integration\.minikube/ca.pem, removing ...
I0915 03:44:08.420621 1756 exec_runner.go:208] rm: C:\Users\jenkins\minikube-integration\.minikube\ca.pem
I0915 03:44:08.420621 1756 exec_runner.go:152] cp: C:\Users\jenkins\minikube-integration\.minikube\certs\ca.pem --> C:\Users\jenkins\minikube-integration\.minikube/ca.pem (1078 bytes)
I0915 03:44:08.422626 1756 exec_runner.go:145] found C:\Users\jenkins\minikube-integration\.minikube/cert.pem, removing ...
I0915 03:44:08.422626 1756 exec_runner.go:208] rm: C:\Users\jenkins\minikube-integration\.minikube\cert.pem
I0915 03:44:08.422626 1756 exec_runner.go:152] cp: C:\Users\jenkins\minikube-integration\.minikube\certs\cert.pem --> C:\Users\jenkins\minikube-integration\.minikube/cert.pem (1123 bytes)
I0915 03:44:08.423620 1756 provision.go:112] generating server cert: C:\Users\jenkins\minikube-integration\.minikube\machines\server.pem ca-key=C:\Users\jenkins\minikube-integration\.minikube\certs\ca.pem private-key=C:\Users\jenkins\minikube-integration\.minikube\certs\ca-key.pem org=jenkins.kubernetes-upgrade-20210915032703-22140 san=[192.168.76.2 127.0.0.1 localhost 127.0.0.1 minikube kubernetes-upgrade-20210915032703-22140]
I0915 03:44:09.014581 1756 provision.go:172] copyRemoteCerts
I0915 03:44:09.028183 1756 ssh_runner.go:152] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0915 03:44:09.044551 1756 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" kubernetes-upgrade-20210915032703-22140
I0915 03:44:08.159118 50104 pod_ready.go:102] pod "coredns-fb8b8dccf-kkhkd" in "kube-system" namespace has status "Ready":"False"
I0915 03:44:10.252438 50104 pod_ready.go:102] pod "coredns-fb8b8dccf-kkhkd" in "kube-system" namespace has status "Ready":"False"
I0915 03:44:08.414520 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:08.896684 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:09.411423 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:09.902716 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:10.401759 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:11.396529 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:12.398080 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:12.900263 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:09.961998 1756 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:58879 SSHKeyPath:C:\Users\jenkins\minikube-integration\.minikube\machines\kubernetes-upgrade-20210915032703-22140\id_rsa Username:docker}
I0915 03:44:11.208395 1756 ssh_runner.go:192] Completed: sudo mkdir -p /etc/docker /etc/docker /etc/docker: (2.179652s)
I0915 03:44:11.208655 1756 ssh_runner.go:319] scp C:\Users\jenkins\minikube-integration\.minikube\certs\ca.pem --> /etc/docker/ca.pem (1078 bytes)
I0915 03:44:12.740414 1756 ssh_runner.go:319] scp C:\Users\jenkins\minikube-integration\.minikube\machines\server.pem --> /etc/docker/server.pem (1289 bytes)
I0915 03:44:13.733988 1756 ssh_runner.go:319] scp C:\Users\jenkins\minikube-integration\.minikube\machines\server-key.pem --> /etc/docker/server-key.pem (1675 bytes)
I0915 03:44:14.463868 1756 provision.go:86] duration metric: configureAuth took 6.8727314s
I0915 03:44:14.463868 1756 ubuntu.go:193] setting minikube options for container-runtime
I0915 03:44:14.464774 1756 config.go:177] Loaded profile config "kubernetes-upgrade-20210915032703-22140": Driver=docker, ContainerRuntime=docker, KubernetesVersion=v1.22.2-rc.0
I0915 03:44:14.474984 1756 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" kubernetes-upgrade-20210915032703-22140
I0915 03:44:12.690601 50104 pod_ready.go:102] pod "coredns-fb8b8dccf-kkhkd" in "kube-system" namespace has status "Ready":"False"
I0915 03:44:15.085391 50104 pod_ready.go:102] pod "coredns-fb8b8dccf-kkhkd" in "kube-system" namespace has status "Ready":"False"
I0915 03:44:13.406216 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:13.900518 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:14.399727 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:14.898891 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:15.410130 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:16.398867 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:16.898823 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:17.907042 38996 ssh_runner.go:152] Run: sudo pgrep -xnf kube-apiserver.*minikube.*
I0915 03:44:15.304811 1756 main.go:130] libmachine: Using SSH client type: native
I0915 03:44:15.304811 1756 main.go:130] libmachine: &{{{<nil> 0 [] [] []} docker [0x158acc0] 0x158db80 <nil> [] 0s} 127.0.0.1 58879 <nil> <nil>}
I0915 03:44:15.304811 1756 main.go:130] libmachine: About to run SSH command:
df --output=fstype / | tail -n 1
I0915 03:44:17.573161 1756 main.go:130] libmachine: SSH cmd err, output: <nil>: overlay
I0915 03:44:17.573161 1756 ubuntu.go:71] root file system type: overlay
I0915 03:44:17.574162 1756 provision.go:309] Updating docker unit: /lib/systemd/system/docker.service ...
I0915 03:44:17.583142 1756 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" kubernetes-upgrade-20210915032703-22140
I0915 03:44:18.416434 1756 main.go:130] libmachine: Using SSH client type: native
I0915 03:44:18.417217 1756 main.go:130] libmachine: &{{{<nil> 0 [] [] []} docker [0x158acc0] 0x158db80 <nil> [] 0s} 127.0.0.1 58879 <nil> <nil>}
I0915 03:44:18.417217 1756 main.go:130] libmachine: About to run SSH command:
sudo mkdir -p /lib/systemd/system && printf %!s(MISSING) "[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60
[Service]
Type=notify
Restart=on-failure
# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
# The base configuration already specifies an 'ExecStart=...' command. The first directive
# here is to clear out that command inherited from the base configuration. Without this,
# the command from the base configuration and the command specified here are treated as
# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
# will catch this invalid input and refuse to start the service with an error like:
# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP \$MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service.new
*
* ==> Docker <==
* -- Logs begin at Wed 2021-09-15 03:36:46 UTC, end at Wed 2021-09-15 03:44:42 UTC. --
Sep 15 03:41:10 cert-options-20210915033619-22140 dockerd[467]: time="2021-09-15T03:41:10.788002700Z" level=info msg="Processing signal 'terminated'"
Sep 15 03:41:10 cert-options-20210915033619-22140 dockerd[467]: time="2021-09-15T03:41:10.850020900Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Sep 15 03:41:10 cert-options-20210915033619-22140 dockerd[467]: time="2021-09-15T03:41:10.862041500Z" level=info msg="Daemon shutdown complete"
Sep 15 03:41:10 cert-options-20210915033619-22140 systemd[1]: docker.service: Succeeded.
Sep 15 03:41:10 cert-options-20210915033619-22140 systemd[1]: Stopped Docker Application Container Engine.
Sep 15 03:41:10 cert-options-20210915033619-22140 systemd[1]: Starting Docker Application Container Engine...
Sep 15 03:41:11 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:11.387850600Z" level=info msg="Starting up"
Sep 15 03:41:11 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:11.398818800Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Sep 15 03:41:11 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:11.399064500Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Sep 15 03:41:11 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:11.400034600Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Sep 15 03:41:11 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:11.400579000Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 15 03:41:11 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:11.415935400Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Sep 15 03:41:11 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:11.434653200Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Sep 15 03:41:11 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:11.434835600Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Sep 15 03:41:11 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:11.434890800Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Sep 15 03:41:11 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:11.899922500Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
Sep 15 03:41:11 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:11.957223300Z" level=info msg="Loading containers: start."
Sep 15 03:41:13 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:13.331752000Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Sep 15 03:41:13 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:13.737835900Z" level=info msg="Loading containers: done."
Sep 15 03:41:14 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:14.150061100Z" level=info msg="Docker daemon" commit=75249d8 graphdriver(s)=overlay2 version=20.10.8
Sep 15 03:41:14 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:14.150232300Z" level=info msg="Daemon has completed initialization"
Sep 15 03:41:14 cert-options-20210915033619-22140 systemd[1]: Started Docker Application Container Engine.
Sep 15 03:41:14 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:14.534127900Z" level=info msg="API listen on [::]:2376"
Sep 15 03:41:14 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:41:14.595268200Z" level=info msg="API listen on /var/run/docker.sock"
Sep 15 03:42:57 cert-options-20210915033619-22140 dockerd[780]: time="2021-09-15T03:42:57.181207300Z" level=info msg="ignoring event" container=209e058816f3e49fc64171a45b298977af28682c787026cbd095239584e8cbeb module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
*
* ==> container status <==
* CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
e5996f04b8a75 8d147537fb7d1 6 seconds ago Running coredns 0 f33854740b76e
a6913a4c819db 6e38f40d628db 18 seconds ago Running storage-provisioner 0 06287b4414aca
fcbd6374c205c 36c4ebbc9d979 24 seconds ago Running kube-proxy 0 709551a17e4e4
62082c6315ab4 6e002eb89a881 About a minute ago Running kube-controller-manager 1 048cf9938fe0e
70217788e6600 0048118155842 2 minutes ago Running etcd 0 1fdf230e30351
f1ec2d6b500ff aca5ededae9c8 2 minutes ago Running kube-scheduler 0 048e90480b461
209e058816f3e 6e002eb89a881 2 minutes ago Exited kube-controller-manager 0 048cf9938fe0e
800b3d462925d f30469a2491a5 2 minutes ago Running kube-apiserver 0 4e27824996523
*
* ==> describe nodes <==
* Name: cert-options-20210915033619-22140
Roles: control-plane,master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=cert-options-20210915033619-22140
kubernetes.io/os=linux
minikube.k8s.io/commit=7d234465a435c40d154c10f5ac847cc10f4e5fc3
minikube.k8s.io/name=cert-options-20210915033619-22140
minikube.k8s.io/updated_at=2021_09_15T03_43_17_0700
minikube.k8s.io/version=v1.23.0
node-role.kubernetes.io/control-plane=
node-role.kubernetes.io/master=
node.kubernetes.io/exclude-from-external-load-balancers=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 15 Sep 2021 03:42:56 +0000
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: cert-options-20210915033619-22140
AcquireTime: <unset>
RenewTime: Wed, 15 Sep 2021 03:44:44 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Wed, 15 Sep 2021 03:44:00 +0000 Wed, 15 Sep 2021 03:42:56 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Wed, 15 Sep 2021 03:44:00 +0000 Wed, 15 Sep 2021 03:42:56 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Wed, 15 Sep 2021 03:44:00 +0000 Wed, 15 Sep 2021 03:42:56 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Wed, 15 Sep 2021 03:44:00 +0000 Wed, 15 Sep 2021 03:44:00 +0000 KubeletReady kubelet is posting ready status
Addresses:
InternalIP: 192.168.58.2
Hostname: cert-options-20210915033619-22140
Capacity:
cpu: 4
ephemeral-storage: 65792556Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 20481980Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 65792556Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 20481980Ki
pods: 110
System Info:
Machine ID: 4b5e5cdd53d44f5ab575bb522d42acca
System UUID: 5e7d2334-11f4-4d32-a0ea-eef70b1c7dc1
Boot ID: 31a72c78-717c-4979-9c6b-d3a794aac31d
Kernel Version: 4.19.121-linuxkit
OS Image: Ubuntu 20.04.2 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://20.10.8
Kubelet Version: v1.22.1
Kube-Proxy Version: v1.22.1
PodCIDR: 10.244.0.0/24
PodCIDRs: 10.244.0.0/24
Non-terminated Pods: (7 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system coredns-78fcd69978-cb8sj 100m (2%!)(MISSING) 0 (0%!)(MISSING) 70Mi (0%!)(MISSING) 170Mi (0%!)(MISSING) 62s
kube-system etcd-cert-options-20210915033619-22140 100m (2%!)(MISSING) 0 (0%!)(MISSING) 100Mi (0%!)(MISSING) 0 (0%!)(MISSING) 50s
kube-system kube-apiserver-cert-options-20210915033619-22140 250m (6%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 50s
kube-system kube-controller-manager-cert-options-20210915033619-22140 200m (5%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 100s
kube-system kube-proxy-wmzsf 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 62s
kube-system kube-scheduler-cert-options-20210915033619-22140 100m (2%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 103s
kube-system storage-provisioner 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 0 (0%!)(MISSING) 46s
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 750m (18%!)(MISSING) 0 (0%!)(MISSING)
memory 170Mi (0%!)(MISSING) 170Mi (0%!)(MISSING)
ephemeral-storage 0 (0%!)(MISSING) 0 (0%!)(MISSING)
hugepages-1Gi 0 (0%!)(MISSING) 0 (0%!)(MISSING)
hugepages-2Mi 0 (0%!)(MISSING) 0 (0%!)(MISSING)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Starting 79s kubelet Starting kubelet.
Normal NodeHasSufficientMemory 74s kubelet Node cert-options-20210915033619-22140 status is now: NodeHasSufficientMemory
Normal NodeHasNoDiskPressure 74s kubelet Node cert-options-20210915033619-22140 status is now: NodeHasNoDiskPressure
Normal NodeHasSufficientPID 74s kubelet Node cert-options-20210915033619-22140 status is now: NodeHasSufficientPID
Normal NodeNotReady 70s kubelet Node cert-options-20210915033619-22140 status is now: NodeNotReady
Normal NodeAllocatableEnforced 53s kubelet Updated Node Allocatable limit across pods
Normal NodeReady 48s kubelet Node cert-options-20210915033619-22140 status is now: NodeReady
*
* ==> dmesg <==
* [ +0.000002] ? ktime_get_update_offsets_now+0x36/0x95
[ +0.000002] hrtimer_interrupt+0x92/0x165
[ +0.000003] hv_stimer0_isr+0x20/0x2d
[ +0.000007] hv_stimer0_vector_handler+0x3b/0x57
[ +0.000009] hv_stimer0_callback_vector+0xf/0x20
[ +0.000001] </IRQ>
[ +0.000001] RIP: 0010:native_safe_halt+0x7/0x8
[ +0.000002] Code: 60 02 df f0 83 44 24 fc 00 48 8b 00 a8 08 74 0b 65 81 25 fd b5 6f 69 ff ff ff 7f c3 e8 77 ce 72 ff f4 c3 e8 70 ce 72 ff fb f4 <c3> 0f 1f 44 00 00 53 e8 f1 f5 81 ff 65 8b 35 b3 4b 6f 69 31 ff e8
[ +0.000001] RSP: 0018:ffff98b6000a3ec8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff12
[ +0.000001] RAX: ffffffff9691a410 RBX: 0000000000000001 RCX: ffffffff97253150
[ +0.000001] RDX: 00000000001bfb3e RSI: 0000000000000001 RDI: 0000000000000001
[ +0.000001] RBP: 0000000000000000 R08: 011cf099150136ab R09: 0000000000000002
[ +0.000000] R10: ffff8b9f6df73938 R11: 0000000000000000 R12: 0000000000000000
[ +0.000001] R13: ffff8b9fae19e1c0 R14: 0000000000000000 R15: 0000000000000000
[ +0.000002] ? ldsem_down_write+0x1da/0x1da
[ +0.000009] ? native_safe_halt+0x5/0x8
[ +0.000001] default_idle+0x1b/0x2c
[ +0.000001] do_idle+0xe5/0x216
[ +0.000002] cpu_startup_entry+0x6f/0x71
[ +0.000003] start_secondary+0x18e/0x1a9
[ +0.000006] secondary_startup_64+0xa4/0xb0
[ +0.000005] ---[ end trace f027fbf82db24e21 ]---
[Sep15 03:23] overlayfs: upperdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
[ +0.000013] overlayfs: workdir is in-use as upperdir/workdir of another mount, accessing files from both mounts will result in undefined behavior.
[Sep15 03:36] tee (174639): /proc/173352/oom_adj is deprecated, please use /proc/173352/oom_score_adj instead.
*
* ==> etcd [70217788e660] <==
* {"level":"info","ts":"2021-09-15T03:43:44.629Z","caller":"traceutil/trace.go:171","msg":"trace[1872656818] transaction","detail":"{read_only:false; response_revision:380; number_of_response:1; }","duration":"148.3519ms","start":"2021-09-15T03:43:44.481Z","end":"2021-09-15T03:43:44.629Z","steps":["trace[1872656818] 'compare' (duration: 29.9237ms)"],"step_count":1}
{"level":"info","ts":"2021-09-15T03:43:44.739Z","caller":"traceutil/trace.go:171","msg":"trace[428826018] transaction","detail":"{read_only:false; response_revision:381; number_of_response:1; }","duration":"158.2293ms","start":"2021-09-15T03:43:44.580Z","end":"2021-09-15T03:43:44.738Z","steps":["trace[428826018] 'process raft request' (duration: 67.3882ms)","trace[428826018] 'compare' (duration: 83.0693ms)"],"step_count":2}
{"level":"info","ts":"2021-09-15T03:43:46.384Z","caller":"traceutil/trace.go:171","msg":"trace[817616794] transaction","detail":"{read_only:false; response_revision:400; number_of_response:1; }","duration":"112.8631ms","start":"2021-09-15T03:43:46.271Z","end":"2021-09-15T03:43:46.384Z","steps":["trace[817616794] 'process raft request' (duration: 22.986ms)","trace[817616794] 'compare' (duration: 83.2916ms)"],"step_count":2}
{"level":"info","ts":"2021-09-15T03:43:46.578Z","caller":"traceutil/trace.go:171","msg":"trace[655274222] transaction","detail":"{read_only:false; response_revision:401; number_of_response:1; }","duration":"114.4558ms","start":"2021-09-15T03:43:46.464Z","end":"2021-09-15T03:43:46.578Z","steps":["trace[655274222] 'compare' (duration: 87.4001ms)"],"step_count":1}
{"level":"info","ts":"2021-09-15T03:43:46.585Z","caller":"traceutil/trace.go:171","msg":"trace[942925476] transaction","detail":"{read_only:false; response_revision:402; number_of_response:1; }","duration":"112.1478ms","start":"2021-09-15T03:43:46.473Z","end":"2021-09-15T03:43:46.585Z","steps":["trace[942925476] 'process raft request' (duration: 86.8865ms)"],"step_count":1}
{"level":"warn","ts":"2021-09-15T03:43:46.647Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"167.0023ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/resourcequotas/default/\" range_end:\"/registry/resourcequotas/default0\" ","response":"range_response_count:0 size:5"}
{"level":"info","ts":"2021-09-15T03:43:46.647Z","caller":"traceutil/trace.go:171","msg":"trace[2069424389] range","detail":"{range_begin:/registry/resourcequotas/default/; range_end:/registry/resourcequotas/default0; response_count:0; response_revision:403; }","duration":"167.1423ms","start":"2021-09-15T03:43:46.480Z","end":"2021-09-15T03:43:46.647Z","steps":["trace[2069424389] 'agreement among raft nodes before linearized reading' (duration: 166.9443ms)"],"step_count":1}
{"level":"warn","ts":"2021-09-15T03:43:46.648Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"127.7203ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/deployments/kube-system/coredns\" ","response":"range_response_count:1 size:3646"}
{"level":"info","ts":"2021-09-15T03:43:46.648Z","caller":"traceutil/trace.go:171","msg":"trace[1777560954] range","detail":"{range_begin:/registry/deployments/kube-system/coredns; range_end:; response_count:1; response_revision:403; }","duration":"127.7664ms","start":"2021-09-15T03:43:46.521Z","end":"2021-09-15T03:43:46.648Z","steps":["trace[1777560954] 'agreement among raft nodes before linearized reading' (duration: 127.6726ms)"],"step_count":1}
{"level":"info","ts":"2021-09-15T03:43:46.906Z","caller":"traceutil/trace.go:171","msg":"trace[1329096700] transaction","detail":"{read_only:false; response_revision:406; number_of_response:1; }","duration":"180.3744ms","start":"2021-09-15T03:43:46.726Z","end":"2021-09-15T03:43:46.906Z","steps":["trace[1329096700] 'process raft request' (duration: 170.8763ms)"],"step_count":1}
{"level":"info","ts":"2021-09-15T03:43:46.918Z","caller":"traceutil/trace.go:171","msg":"trace[1318333876] transaction","detail":"{read_only:false; response_revision:405; number_of_response:1; }","duration":"192.1033ms","start":"2021-09-15T03:43:46.725Z","end":"2021-09-15T03:43:46.918Z","steps":["trace[1318333876] 'process raft request' (duration: 170.7883ms)"],"step_count":1}
{"level":"info","ts":"2021-09-15T03:43:47.337Z","caller":"traceutil/trace.go:171","msg":"trace[192655157] transaction","detail":"{read_only:false; response_revision:420; number_of_response:1; }","duration":"102.2212ms","start":"2021-09-15T03:43:47.235Z","end":"2021-09-15T03:43:47.337Z","steps":["trace[192655157] 'process raft request' (duration: 50.7742ms)"],"step_count":1}
{"level":"info","ts":"2021-09-15T03:43:47.346Z","caller":"traceutil/trace.go:171","msg":"trace[150370285] transaction","detail":"{read_only:false; response_revision:421; number_of_response:1; }","duration":"110.6456ms","start":"2021-09-15T03:43:47.235Z","end":"2021-09-15T03:43:47.346Z","steps":["trace[150370285] 'process raft request' (duration: 100.515ms)"],"step_count":1}
{"level":"info","ts":"2021-09-15T03:43:58.941Z","caller":"traceutil/trace.go:171","msg":"trace[245569927] transaction","detail":"{read_only:false; response_revision:435; number_of_response:1; }","duration":"116.1234ms","start":"2021-09-15T03:43:58.824Z","end":"2021-09-15T03:43:58.941Z","steps":["trace[245569927] 'process raft request' (duration: 40.7001ms)"],"step_count":1}
{"level":"info","ts":"2021-09-15T03:43:58.960Z","caller":"traceutil/trace.go:171","msg":"trace[54918223] linearizableReadLoop","detail":"{readStateIndex:450; appliedIndex:450; }","duration":"100.8414ms","start":"2021-09-15T03:43:58.859Z","end":"2021-09-15T03:43:58.960Z","steps":["trace[54918223] 'read index received' (duration: 100.8295ms)","trace[54918223] 'applied index is now lower than readState.Index' (duration: 10.3µs)"],"step_count":2}
{"level":"warn","ts":"2021-09-15T03:43:59.039Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"180.4229ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/serviceaccounts/kube-system/kube-proxy\" ","response":"range_response_count:1 size:226"}
{"level":"info","ts":"2021-09-15T03:43:59.039Z","caller":"traceutil/trace.go:171","msg":"trace[1832696591] range","detail":"{range_begin:/registry/serviceaccounts/kube-system/kube-proxy; range_end:; response_count:1; response_revision:435; }","duration":"180.6791ms","start":"2021-09-15T03:43:58.859Z","end":"2021-09-15T03:43:59.039Z","steps":["trace[1832696591] 'agreement among raft nodes before linearized reading' (duration: 101.2687ms)","trace[1832696591] 'range keys from in-memory index tree' (duration: 79.2337ms)"],"step_count":2}
{"level":"info","ts":"2021-09-15T03:44:01.118Z","caller":"traceutil/trace.go:171","msg":"trace[1296453845] linearizableReadLoop","detail":"{readStateIndex:459; appliedIndex:459; }","duration":"117.66ms","start":"2021-09-15T03:44:01.000Z","end":"2021-09-15T03:44:01.117Z","steps":["trace[1296453845] 'read index received' (duration: 117.6488ms)","trace[1296453845] 'applied index is now lower than readState.Index' (duration: 9.3µs)"],"step_count":2}
{"level":"warn","ts":"2021-09-15T03:44:01.136Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"136.4855ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/pods/kube-system/kube-scheduler-cert-options-20210915033619-22140\" ","response":"range_response_count:1 size:4357"}
{"level":"info","ts":"2021-09-15T03:44:01.136Z","caller":"traceutil/trace.go:171","msg":"trace[2127601156] range","detail":"{range_begin:/registry/pods/kube-system/kube-scheduler-cert-options-20210915033619-22140; range_end:; response_count:1; response_revision:443; }","duration":"136.6165ms","start":"2021-09-15T03:44:01.000Z","end":"2021-09-15T03:44:01.136Z","steps":["trace[2127601156] 'agreement among raft nodes before linearized reading' (duration: 118.1878ms)","trace[2127601156] 'range keys from in-memory index tree' (duration: 18.2211ms)"],"step_count":2}
{"level":"info","ts":"2021-09-15T03:44:01.145Z","caller":"traceutil/trace.go:171","msg":"trace[1799665743] transaction","detail":"{read_only:false; response_revision:444; number_of_response:1; }","duration":"121.4429ms","start":"2021-09-15T03:44:01.023Z","end":"2021-09-15T03:44:01.145Z","steps":["trace[1799665743] 'process raft request' (duration: 94.419ms)","trace[1799665743] 'compare' (duration: 17.9865ms)"],"step_count":2}
{"level":"info","ts":"2021-09-15T03:44:18.759Z","caller":"traceutil/trace.go:171","msg":"trace[1036487153] transaction","detail":"{read_only:false; response_revision:466; number_of_response:1; }","duration":"135.1351ms","start":"2021-09-15T03:44:18.624Z","end":"2021-09-15T03:44:18.759Z","steps":["trace[1036487153] 'process raft request' (duration: 111.8141ms)"],"step_count":1}
{"level":"info","ts":"2021-09-15T03:44:20.928Z","caller":"traceutil/trace.go:171","msg":"trace[1945002690] linearizableReadLoop","detail":"{readStateIndex:490; appliedIndex:490; }","duration":"120.8154ms","start":"2021-09-15T03:44:20.807Z","end":"2021-09-15T03:44:20.928Z","steps":["trace[1945002690] 'read index received' (duration: 120.8037ms)","trace[1945002690] 'applied index is now lower than readState.Index' (duration: 9.7µs)"],"step_count":2}
{"level":"warn","ts":"2021-09-15T03:44:20.987Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"178.9965ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/pods/kube-system/kube-scheduler-cert-options-20210915033619-22140\" ","response":"range_response_count:1 size:4548"}
{"level":"info","ts":"2021-09-15T03:44:20.987Z","caller":"traceutil/trace.go:171","msg":"trace[586625846] range","detail":"{range_begin:/registry/pods/kube-system/kube-scheduler-cert-options-20210915033619-22140; range_end:; response_count:1; response_revision:467; }","duration":"179.7594ms","start":"2021-09-15T03:44:20.807Z","end":"2021-09-15T03:44:20.987Z","steps":["trace[586625846] 'agreement among raft nodes before linearized reading' (duration: 121.0952ms)","trace[586625846] 'range keys from in-memory index tree' (duration: 57.9806ms)"],"step_count":2}
*
* ==> kernel <==
* 03:44:50 up 2:21, 0 users, load average: 60.62, 35.04, 24.65
Linux cert-options-20210915033619-22140 4.19.121-linuxkit #1 SMP Thu Jan 21 15:36:34 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 20.04.2 LTS"
*
* ==> kube-apiserver [800b3d462925] <==
* Trace[1700767156]: ---"Object stored in database" 683ms (03:43:44.946)
Trace[1700767156]: [822.1098ms] [822.1098ms] END
I0915 03:43:44.999600 1 trace.go:205] Trace[866074677]: "Update" url:/api/v1/namespaces/kube-system/pods/kube-scheduler-cert-options-20210915033619-22140/status,user-agent:kube-controller-manager/v1.22.1 (linux/amd64) kubernetes/632ed30/system:serviceaccount:kube-system:node-controller,audit-id:e1e5c7c9-1407-4188-8085-79172a6973fb,client:192.168.58.2,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-Sep-2021 03:43:44.335) (total time: 660ms):
Trace[866074677]: ---"Object stored in database" 602ms (03:43:44.938)
Trace[866074677]: [660.6894ms] [660.6894ms] END
I0915 03:43:45.204085 1 trace.go:205] Trace[632892554]: "Get" url:/api/v1/namespaces/kube-system/serviceaccounts/bootstrap-signer,user-agent:kube-controller-manager/v1.22.1 (linux/amd64) kubernetes/632ed30/kube-controller-manager,audit-id:d45996e3-d83c-44a0-bdea-f8dde225f0cf,client:192.168.58.2,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-Sep-2021 03:43:44.682) (total time: 521ms):
Trace[632892554]: ---"About to write a response" 521ms (03:43:45.203)
Trace[632892554]: [521.9813ms] [521.9813ms] END
I0915 03:43:45.749795 1 controller.go:611] quota admission added evaluator for: replicasets.apps
I0915 03:43:46.106808 1 controller.go:611] quota admission added evaluator for: controllerrevisions.apps
I0915 03:43:46.852925 1 trace.go:205] Trace[325309241]: "Create" url:/api/v1/namespaces/kube-system/pods,user-agent:kube-controller-manager/v1.22.1 (linux/amd64) kubernetes/632ed30/system:serviceaccount:kube-system:daemon-set-controller,audit-id:1c4f647f-490d-4891-b85f-075ac6366db2,client:192.168.58.2,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-Sep-2021 03:43:46.311) (total time: 541ms):
Trace[325309241]: ---"Object stored in database" 516ms (03:43:46.852)
Trace[325309241]: [541.7425ms] [541.7425ms] END
I0915 03:43:46.908967 1 trace.go:205] Trace[2135736814]: "Create" url:/api/v1/namespaces/default/serviceaccounts,user-agent:kube-controller-manager/v1.22.1 (linux/amd64) kubernetes/632ed30/system:serviceaccount:kube-system:service-account-controller,audit-id:17f7d653-edbd-4ae4-9f8b-7eb5125141bb,client:192.168.58.2,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-Sep-2021 03:43:46.309) (total time: 599ms):
Trace[2135736814]: ---"Object stored in database" 599ms (03:43:46.908)
Trace[2135736814]: [599.6866ms] [599.6866ms] END
I0915 03:43:46.967776 1 trace.go:205] Trace[399309015]: "Create" url:/api/v1/namespaces/default/configmaps,user-agent:kube-controller-manager/v1.22.1 (linux/amd64) kubernetes/632ed30/system:serviceaccount:kube-system:root-ca-cert-publisher,audit-id:c42b8a12-279a-4976-be00-0f3b942f9cff,client:192.168.58.2,accept:application/vnd.kubernetes.protobuf, */*,protocol:HTTP/2.0 (15-Sep-2021 03:43:46.404) (total time: 563ms):
Trace[399309015]: ---"Object stored in database" 562ms (03:43:46.967)
Trace[399309015]: [563.0785ms] [563.0785ms] END
I0915 03:43:59.047325 1 trace.go:205] Trace[558591707]: "Create" url:/api/v1/namespaces/kube-system/pods,user-agent:kubelet/v1.22.1 (linux/amd64) kubernetes/632ed30,audit-id:f494911e-202a-41f1-8ed6-6fdb546ec780,client:192.168.58.2,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (15-Sep-2021 03:43:58.055) (total time: 991ms):
Trace[558591707]: ---"Object stored in database" 943ms (03:43:59.020)
Trace[558591707]: [991.2937ms] [991.2937ms] END
I0915 03:43:59.067549 1 trace.go:205] Trace[1890453055]: "Create" url:/api/v1/namespaces/kube-system/pods,user-agent:kubelet/v1.22.1 (linux/amd64) kubernetes/632ed30,audit-id:b5e17540-7842-4096-960f-a4b2843ff538,client:192.168.58.2,accept:application/vnd.kubernetes.protobuf,application/json,protocol:HTTP/2.0 (15-Sep-2021 03:43:58.119) (total time: 908ms):
Trace[1890453055]: ---"Object stored in database" 897ms (03:43:59.027)
Trace[1890453055]: [908.0016ms] [908.0016ms] END
*
* ==> kube-controller-manager [209e058816f3] <==
* /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00037e490, 0x5175b80, 0xc000ce0120, 0x4c62201, 0xc000094360)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0x9b
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00037e490, 0x3b9aca00, 0x0, 0x1, 0xc000094360)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc00037e490, 0x3b9aca00, 0xc000094360)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicServingCertificateController).Run
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/tlsconfig.go:247 +0x1d2
goroutine 180 [select]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00037e530, 0x5175b80, 0xc000ce00f0, 0x1, 0xc000094360)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:167 +0x118
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00037e530, 0xdf8475800, 0x0, 0x1, 0xc000094360)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc00037e530, 0xdf8475800, 0xc000094360)
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x4d
created by k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates.(*DynamicServingCertificateController).Run
/workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apiserver/pkg/server/dynamiccertificates/tlsconfig.go:250 +0x24b
goroutine 155 [runnable]:
net/http.setRequestCancel.func4(0x0, 0xc000aadc20, 0xc0002e75e0, 0xc000c9860c, 0xc00043b260)
/usr/local/go/src/net/http/client.go:397 +0x96
created by net/http.setRequestCancel
/usr/local/go/src/net/http/client.go:396 +0x337
*
* ==> kube-controller-manager [62082c6315ab] <==
* I0915 03:43:43.959939 1 event.go:291] "Event occurred" object="cert-options-20210915033619-22140" kind="Node" apiVersion="v1" type="Normal" reason="RegisteredNode" message="Node cert-options-20210915033619-22140 event: Registered Node cert-options-20210915033619-22140 in Controller"
I0915 03:43:43.999092 1 shared_informer.go:247] Caches are synced for ReplicaSet
I0915 03:43:44.002191 1 shared_informer.go:247] Caches are synced for namespace
I0915 03:43:44.031423 1 shared_informer.go:247] Caches are synced for disruption
I0915 03:43:44.031489 1 disruption.go:371] Sending events to api server.
I0915 03:43:44.059934 1 shared_informer.go:247] Caches are synced for deployment
I0915 03:43:44.575453 1 shared_informer.go:247] Caches are synced for crt configmap
I0915 03:43:44.604483 1 shared_informer.go:247] Caches are synced for stateful set
I0915 03:43:44.610464 1 shared_informer.go:247] Caches are synced for bootstrap_signer
I0915 03:43:44.611164 1 shared_informer.go:247] Caches are synced for expand
I0915 03:43:44.677434 1 shared_informer.go:247] Caches are synced for PVC protection
I0915 03:43:44.692758 1 shared_informer.go:247] Caches are synced for persistent volume
I0915 03:43:44.721033 1 shared_informer.go:247] Caches are synced for resource quota
I0915 03:43:44.721079 1 shared_informer.go:247] Caches are synced for attach detach
I0915 03:43:44.732398 1 shared_informer.go:247] Caches are synced for resource quota
I0915 03:43:44.732501 1 shared_informer.go:247] Caches are synced for ephemeral
I0915 03:43:44.927059 1 shared_informer.go:240] Waiting for caches to sync for garbage collector
I0915 03:43:45.014736 1 event.go:291] "Event occurred" object="kube-system/kube-scheduler-cert-options-20210915033619-22140" kind="Pod" apiVersion="v1" type="Warning" reason="NodeNotReady" message="Node is not ready"
I0915 03:43:45.541671 1 shared_informer.go:247] Caches are synced for garbage collector
I0915 03:43:45.562920 1 shared_informer.go:247] Caches are synced for garbage collector
I0915 03:43:45.562947 1 garbagecollector.go:151] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
I0915 03:43:45.912912 1 event.go:291] "Event occurred" object="kube-system/coredns" kind="Deployment" apiVersion="apps/v1" type="Normal" reason="ScalingReplicaSet" message="Scaled up replica set coredns-78fcd69978 to 1"
I0915 03:43:46.864169 1 event.go:291] "Event occurred" object="kube-system/kube-proxy" kind="DaemonSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: kube-proxy-wmzsf"
I0915 03:43:46.877071 1 event.go:291] "Event occurred" object="kube-system/coredns-78fcd69978" kind="ReplicaSet" apiVersion="apps/v1" type="Normal" reason="SuccessfulCreate" message="Created pod: coredns-78fcd69978-cb8sj"
I0915 03:44:03.987780 1 node_lifecycle_controller.go:1191] Controller detected that some Nodes are Ready. Exiting master disruption mode.
*
* ==> kube-proxy [fcbd6374c205] <==
* I0915 03:44:33.599531 1 node.go:172] Successfully retrieved node IP: 192.168.58.2
I0915 03:44:33.599880 1 server_others.go:140] Detected node IP 192.168.58.2
W0915 03:44:33.601197 1 server_others.go:565] Unknown proxy mode "", assuming iptables proxy
I0915 03:44:35.281392 1 server_others.go:206] kube-proxy running in dual-stack mode, IPv4-primary
I0915 03:44:35.283106 1 server_others.go:212] Using iptables Proxier.
I0915 03:44:35.283149 1 server_others.go:219] creating dualStackProxier for iptables.
W0915 03:44:35.283205 1 server_others.go:495] detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6
I0915 03:44:35.323273 1 server.go:649] Version: v1.22.1
I0915 03:44:35.339505 1 config.go:315] Starting service config controller
I0915 03:44:35.340275 1 shared_informer.go:240] Waiting for caches to sync for service config
I0915 03:44:35.361464 1 config.go:224] Starting endpoint slice config controller
I0915 03:44:35.361543 1 shared_informer.go:240] Waiting for caches to sync for endpoint slice config
E0915 03:44:35.477163 1 event_broadcaster.go:253] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cert-options-20210915033619-22140.16a4e204d2f9c4a4", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, EventTime:v1.MicroTime{Time:time.Time{wall:0xc04878e8d56da3d4, ext:6002180301, loc:(*time.Location)(0x2d81340)}}, Series:(*v1.EventSeries)(nil), ReportingController:"kube-proxy", ReportingInstance:"kube-proxy-cert-options-20210915033619-22140", Action:"StartKubeProxy", Reason:"Starting", Regarding:v1.ObjectReference{Kind:"Node", Namespace:""
, Name:"cert-options-20210915033619-22140", UID:"cert-options-20210915033619-22140", APIVersion:"", ResourceVersion:"", FieldPath:""}, Related:(*v1.ObjectReference)(nil), Note:"", Type:"Normal", DeprecatedSource:v1.EventSource{Component:"", Host:""}, DeprecatedFirstTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeprecatedLastTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeprecatedCount:0}': 'Event "cert-options-20210915033619-22140.16a4e204d2f9c4a4" is invalid: involvedObject.namespace: Invalid value: "": does not match event.namespace' (will not retry!)
I0915 03:44:35.490660 1 shared_informer.go:247] Caches are synced for service config
I0915 03:44:35.563119 1 shared_informer.go:247] Caches are synced for endpoint slice config
*
* ==> kube-scheduler [f1ec2d6b500f] <==
* E0915 03:42:58.553515 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicationController: failed to list *v1.ReplicationController: replicationcontrollers is forbidden: User "system:kube-scheduler" cannot list resource "replicationcontrollers" in API group "" at the cluster scope
E0915 03:42:58.583680 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0915 03:42:58.828151 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:kube-scheduler" cannot list resource "namespaces" in API group "" at the cluster scope
E0915 03:42:58.927033 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0915 03:42:59.194742 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0915 03:42:59.309263 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.CSIStorageCapacity: failed to list *v1beta1.CSIStorageCapacity: csistoragecapacities.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csistoragecapacities" in API group "storage.k8s.io" at the cluster scope
E0915 03:42:59.554069 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0915 03:42:59.595037 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope
E0915 03:42:59.750736 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
E0915 03:42:59.939063 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:kube-scheduler" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0915 03:43:00.075850 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csidrivers" in API group "storage.k8s.io" at the cluster scope
E0915 03:43:00.116749 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:kube-scheduler" cannot list resource "statefulsets" in API group "apps" at the cluster scope
E0915 03:43:00.186781 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: storageclasses.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "storageclasses" in API group "storage.k8s.io" at the cluster scope
E0915 03:43:00.206629 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: services is forbidden: User "system:kube-scheduler" cannot list resource "services" in API group "" at the cluster scope
E0915 03:43:00.260279 1 reflector.go:138] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:205: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
E0915 03:43:03.282048 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PodDisruptionBudget: failed to list *v1.PodDisruptionBudget: poddisruptionbudgets.policy is forbidden: User "system:kube-scheduler" cannot list resource "poddisruptionbudgets" in API group "policy" at the cluster scope
E0915 03:43:03.305837 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User "system:kube-scheduler" cannot list resource "pods" in API group "" at the cluster scope
E0915 03:43:03.776002 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.CSIStorageCapacity: failed to list *v1beta1.CSIStorageCapacity: csistoragecapacities.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csistoragecapacities" in API group "storage.k8s.io" at the cluster scope
E0915 03:43:03.809911 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSINode: failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:kube-scheduler" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope
E0915 03:43:03.975960 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:kube-scheduler" cannot list resource "persistentvolumes" in API group "" at the cluster scope
E0915 03:43:04.520884 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Namespace: failed to list *v1.Namespace: namespaces is forbidden: User "system:kube-scheduler" cannot list resource "namespaces" in API group "" at the cluster scope
E0915 03:43:05.123759 1 reflector.go:138] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:205: Failed to watch *v1.ConfigMap: failed to list *v1.ConfigMap: configmaps "extension-apiserver-authentication" is forbidden: User "system:kube-scheduler" cannot list resource "configmaps" in API group "" in the namespace "kube-system"
E0915 03:43:12.158568 1 plugin.go:138] "getting namespace, assuming empty set of namespace labels" err="namespace \"kube-system\" not found" namespace="kube-system"
E0915 03:43:12.158635 1 plugin.go:138] "getting namespace, assuming empty set of namespace labels" err="namespace \"kube-system\" not found" namespace="kube-system"
I0915 03:43:16.979617 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
*
* ==> kubelet <==
* -- Logs begin at Wed 2021-09-15 03:36:46 UTC, end at Wed 2021-09-15 03:44:59 UTC. --
Sep 15 03:43:58 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:43:58.171948 2762 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-certs\" (UniqueName: \"kubernetes.io/host-path/adf02b8889175b1dced2de1cfa857d0a-ca-certs\") pod \"kube-controller-manager-cert-options-20210915033619-22140\" (UID: \"adf02b8889175b1dced2de1cfa857d0a\") "
Sep 15 03:43:58 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:43:58.172496 2762 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"flexvolume-dir\" (UniqueName: \"kubernetes.io/host-path/adf02b8889175b1dced2de1cfa857d0a-flexvolume-dir\") pod \"kube-controller-manager-cert-options-20210915033619-22140\" (UID: \"adf02b8889175b1dced2de1cfa857d0a\") "
Sep 15 03:43:58 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:43:58.172560 2762 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"k8s-certs\" (UniqueName: \"kubernetes.io/host-path/adf02b8889175b1dced2de1cfa857d0a-k8s-certs\") pod \"kube-controller-manager-cert-options-20210915033619-22140\" (UID: \"adf02b8889175b1dced2de1cfa857d0a\") "
Sep 15 03:43:58 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:43:58.172602 2762 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubeconfig\" (UniqueName: \"kubernetes.io/host-path/f6e7d881d17f6d944d2a54576503290f-kubeconfig\") pod \"kube-scheduler-cert-options-20210915033619-22140\" (UID: \"f6e7d881d17f6d944d2a54576503290f\") "
Sep 15 03:43:58 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:43:58.172640 2762 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-data\" (UniqueName: \"kubernetes.io/host-path/8d680e4b34ed251e9091ff208324aae5-etcd-data\") pod \"etcd-cert-options-20210915033619-22140\" (UID: \"8d680e4b34ed251e9091ff208324aae5\") "
Sep 15 03:43:58 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:43:58.172700 2762 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"usr-local-share-ca-certificates\" (UniqueName: \"kubernetes.io/host-path/4eeccf244425bf6c5c6b6aeb52bab502-usr-local-share-ca-certificates\") pod \"kube-apiserver-cert-options-20210915033619-22140\" (UID: \"4eeccf244425bf6c5c6b6aeb52bab502\") "
Sep 15 03:43:58 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:43:58.172760 2762 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"usr-share-ca-certificates\" (UniqueName: \"kubernetes.io/host-path/adf02b8889175b1dced2de1cfa857d0a-usr-share-ca-certificates\") pod \"kube-controller-manager-cert-options-20210915033619-22140\" (UID: \"adf02b8889175b1dced2de1cfa857d0a\") "
Sep 15 03:43:58 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:43:58.172781 2762 reconciler.go:157] "Reconciler: start to sync state"
Sep 15 03:44:01 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:44:01.040430 2762 status_manager.go:276] "Container startup changed before pod has synced" pod="kube-system/kube-apiserver-cert-options-20210915033619-22140" containerID="docker://800b3d462925d79f6e719f9dfdf286da8a477ec6b9c9fa9341eb375f8b9fce68"
Sep 15 03:44:02 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:44:02.939986 2762 status_manager.go:276] "Container startup changed before pod has synced" pod="kube-system/etcd-cert-options-20210915033619-22140" containerID="docker://70217788e6600e896d2240c58da010916e65b659ed42c715c6c5f4ce67b6647f"
Sep 15 03:44:04 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:44:04.714326 2762 topology_manager.go:200] "Topology Admit Handler"
Sep 15 03:44:04 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:44:04.758575 2762 topology_manager.go:200] "Topology Admit Handler"
Sep 15 03:44:04 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:44:04.809911 2762 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjz2c\" (UniqueName: \"kubernetes.io/projected/42ce7029-cc6b-4bb9-b68b-befa68dc6449-kube-api-access-tjz2c\") pod \"storage-provisioner\" (UID: \"42ce7029-cc6b-4bb9-b68b-befa68dc6449\") "
Sep 15 03:44:04 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:44:04.845495 2762 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8jxv\" (UniqueName: \"kubernetes.io/projected/6c354c34-6a2e-45c2-ab92-5442b51a8b82-kube-api-access-k8jxv\") pod \"coredns-78fcd69978-cb8sj\" (UID: \"6c354c34-6a2e-45c2-ab92-5442b51a8b82\") "
Sep 15 03:44:04 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:44:04.850475 2762 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/host-path/42ce7029-cc6b-4bb9-b68b-befa68dc6449-tmp\") pod \"storage-provisioner\" (UID: \"42ce7029-cc6b-4bb9-b68b-befa68dc6449\") "
Sep 15 03:44:04 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:44:04.850663 2762 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c354c34-6a2e-45c2-ab92-5442b51a8b82-config-volume\") pod \"coredns-78fcd69978-cb8sj\" (UID: \"6c354c34-6a2e-45c2-ab92-5442b51a8b82\") "
Sep 15 03:44:05 cert-options-20210915033619-22140 kubelet[2762]: W0915 03:44:05.214527 2762 container.go:586] Failed to update stats for container "/kubepods/besteffort/pod42ce7029-cc6b-4bb9-b68b-befa68dc6449": /sys/fs/cgroup/cpuset/kubepods/besteffort/pod42ce7029-cc6b-4bb9-b68b-befa68dc6449/cpuset.mems found to be empty, continuing to push stats
Sep 15 03:44:15 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:44:15.185480 2762 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="709551a17e4e44b25190314cb5a266090fe67404accc0dc6eab5522ebd4db870"
Sep 15 03:44:15 cert-options-20210915033619-22140 kubelet[2762]: E0915 03:44:15.746344 2762 kubelet.go:1701] "Failed creating a mirror pod for" err="pods \"kube-apiserver-cert-options-20210915033619-22140\" already exists" pod="kube-system/kube-apiserver-cert-options-20210915033619-22140"
Sep 15 03:44:16 cert-options-20210915033619-22140 kubelet[2762]: E0915 03:44:16.316020 2762 kubelet.go:1701] "Failed creating a mirror pod for" err="pods \"etcd-cert-options-20210915033619-22140\" already exists" pod="kube-system/etcd-cert-options-20210915033619-22140"
Sep 15 03:44:26 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:44:26.331924 2762 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="06287b4414aca5470da58f36fe4523dcbcdb23cb5a31cb4e7544957b1d7186c0"
Sep 15 03:44:37 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:44:37.982673 2762 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for kube-system/coredns-78fcd69978-cb8sj through plugin: invalid network status for"
Sep 15 03:44:38 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:44:38.703361 2762 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="f33854740b76eafcaf6db52e0c195ffa05d12d5ecc0f80c1d52820dd2ce0b921"
Sep 15 03:44:40 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:44:40.904066 2762 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for kube-system/coredns-78fcd69978-cb8sj through plugin: invalid network status for"
Sep 15 03:44:45 cert-options-20210915033619-22140 kubelet[2762]: I0915 03:44:45.879712 2762 docker_sandbox.go:401] "Failed to read pod IP from plugin/docker" err="Couldn't find network status for kube-system/coredns-78fcd69978-cb8sj through plugin: invalid network status for"
*
* ==> storage-provisioner [a6913a4c819d] <==
* I0915 03:44:40.517399 1 storage_provisioner.go:116] Initializing the minikube storage provisioner...
-- /stdout --
** stderr **
! Executing "docker container inspect cert-options-20210915033619-22140 --format={{.State.Status}}" took an unusually long time: 2.3237132s
* Restarting the docker service may improve performance.
** /stderr **
helpers_test.go:255: (dbg) Run: out/minikube-windows-amd64.exe status --format={{.APIServer}} -p cert-options-20210915033619-22140 -n cert-options-20210915033619-22140
helpers_test.go:255: (dbg) Done: out/minikube-windows-amd64.exe status --format={{.APIServer}} -p cert-options-20210915033619-22140 -n cert-options-20210915033619-22140: (6.9896078s)
helpers_test.go:262: (dbg) Run: kubectl --context cert-options-20210915033619-22140 get po -o=jsonpath={.items[*].metadata.name} -A --field-selector=status.phase!=Running
helpers_test.go:262: (dbg) Done: kubectl --context cert-options-20210915033619-22140 get po -o=jsonpath={.items[*].metadata.name} -A --field-selector=status.phase!=Running: (1.0803355s)
helpers_test.go:271: non-running pods:
helpers_test.go:273: ======> post-mortem[TestCertOptions]: describe non-running pods <======
helpers_test.go:276: (dbg) Run: kubectl --context cert-options-20210915033619-22140 describe pod
helpers_test.go:276: (dbg) Non-zero exit: kubectl --context cert-options-20210915033619-22140 describe pod : exit status 1 (256.3534ms)
** stderr **
error: resource name may not be empty
** /stderr **
helpers_test.go:278: kubectl --context cert-options-20210915033619-22140 describe pod : exit status 1
helpers_test.go:176: Cleaning up "cert-options-20210915033619-22140" profile ...
helpers_test.go:179: (dbg) Run: out/minikube-windows-amd64.exe delete -p cert-options-20210915033619-22140
=== CONT TestCertOptions
helpers_test.go:179: (dbg) Done: out/minikube-windows-amd64.exe delete -p cert-options-20210915033619-22140: (33.2657684s)
--- FAIL: TestCertOptions (562.68s)